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Preface 

On  April  20,  1966,  the  National  Bureau  of  Standards  sponsored  a  one-day  symposium  whose  purpose 
was  to  convene  operations  research  practitioners  and  users  in  the  agencies  of ^thVPedeS  ESSSZSStlZ 
that  they  might  exchange  information  on  topics  of  mutual  concern  ^eaerai  government  so 

related  activiSf  COnsiderations  underlay  the  &™mng  of  the  conference,  and  our  hopes  for  future 

(1)  There  does  not  exist,  but  there  should  exist,  a  reference  inventory  of  completed  and  on-eointr 
operations  research/systems  analysis/multidisciplinary  research/scientific   planning  activities  ir f  the 

oltett  ctldTeTctlitatod^  &  °f  meth°dS>  be"efit  -aS^Uh! 

tiona(iaeffSieveenesse  meth°dS  t0  ^  aPPHed  that        n0t  UniqUe  t0  °ne  agency>  e^>  tests  of  organiza- 

(b)  There  are  suggestive  analogies  among  problem  types  from  agency  to  agency,  e  g    a  net- 
work of  highways  and  a  network  of  post  offices.  «-seiicy,  e.g.,  a  net 

(c)  There  are  interaction  situations,  e.g.,  the  full  program  benefit  to  one  agency  may  be  in- 
completely calculated  due  to  excluding  benefits  arising  to  other  Federal  missions 

(2)  The  growth  in  opportunity  to  test  the  usefulness  of  systems  analysis  in  new  non-defense  con- 
texts underscores  the  importance  of  making  sure  that  any  initial  study  has  the  fullest  possible  measure 
of  success  at  an  early  date,  and  this  might  exclude  repeating  exploratory  work  others  have  done  No 
reasumng  encyclopedic  memory  bank  exists. 

(3)  Each  of  us  knows  of  a  dozen  or  two  other  studies  or  groups,  but  the  creation  of  new  e-roups 
is  making  it  more  difficult  and  time-consuming  to  keep  up  to  date.  We  need  a  new  one-time  updating 

Sv J™  <l^ll°iaC*\0lff  th6  n6W  rU?aU  ot-  the  B.udget  PIannin^  Programming,  and  Budgeting 
System  (Circular  66-3)  places  an  explicit  requirement  on  agencies  to  provide  systems  analysis,  binefit- 
cost  studies,  and  supporting  documentation.  Some  agency  chiefs  have  asked  OR  groups,  "Tell  me  what 
fXr^n/  to,heiP;  th,e  °R  ^oups  need  to  meet  with  the  PPB  originators  on  a  technical  basTs  for  a 
fuller  understanding  of  the  substantive  requirements  and  desires  so  that  they  can  help  their  parent 
agency  to  respond  adequately  to  this  new  system.  P  parent 

(5)  The  problem  of  identifying  measures  of  effectiveness  for  agencies  is  a  common  one;  some  have 
made  good  progress  in  this  area,  and  others  are  searching  for  ways  to  make  a  good  start.  Furthermore 
many  agencies  have  missions  whose  ultimate  measures  of  effectiveness  can  be  partly  stated  in  common 
terms,  e.g.,  stimulate  the  economy  improve  the  standard  of  living,  etc.  Aside  from  the  recent  Brookings 
Institution  publications  Roland  McKean's  earlier  book,  the  BOB  bibliography  of  benefit-cost  studies,  and 
a  few  other  attempts  of  a  more  nebulous  nature,  the  best  information  on  measures  of  benefit  is  scattered 
among  our  individual  heads  and  in  individual  files.  scduueiea 

(6)  The  changing  semantics  and  connotative  nuances  placed  on  the  terms,  "operations  research  " 
systems  analysis,      multidisciplinary  problem-solving  teams,"  and  "planning  systems,"  are  further 

confused  by  actual  practice.  For  example,  OR  is  frequently  defined  to  be  identical  with  what  the  statis- 
tics group  does,  or  with  the  problems  that  are  sent  to  the  computing  lab,  or  with  the  management 
analysis  program  of  an  agency,  or  with  just  plain  horse  sense.  Accordingly,  there  does  not  seem  to  be 
a  good  common  dictionary  of  what  to  include,  or  what  to  exclude  from  a  conference  such  as  this  Each 
agency  has  one  or  more  groups  with  something  significant  to  contribute  or  gain;  these  groups  are  all 
talking,  with  greater  or  lesser  degree  of  specialization,  to  a  common  problem— that  of  providing  a  ra- 
tional scheme  for  identifying  and  comparing  alternatives  in  terms  of  their  expected  payoff  and  resource 
requirements.  Similar  differences  exist  in  the  dimension  that  starts  with  pure  methods  research  and  ends 
somewhere  with  "the  back  of  the  envelope  calculation"  or  "the  horseback  guess."  We  thought  it  better  to 
invite  all  these  audiences  in  the  first  round,  and  let  the  agency  decide  whether  the  conference  was  rele- 
vant enough  to  participate. 

(7)  With  non-defense  OR  getting  a  stronger  impetus,  there  is  an  obvious  need  to  profit  from  the 
lessons  learned  m  the  Defense  business.  There  is  also  the  need  to  make  sure  that  when  defense  readiness 
is  a  relevant  measure  of  effectiveness  of  civilian  programs,  it  is  included;  the  reverse  is  also  true 
Furthermore,  because  of  the  scarcity  of  professional  personnel  at  the  present  time,  it  is  important  that 
the  growth  of  non-Defense  OR  not  compromise  Defense  requirements. 

(8)  Procedural  and  institutional  problems  related  to  starting  and  effectively  using  OR  groups 
probably  have  a  large  degree  of  similarity  from  agency  to  agency.  The  lone  systems  analyst  who  has 
been  instructed  to  prepare  a  plan  to  start  a  viable  activity  in  the  agency  should  have  access  to  plans 
that  have  a  good  chance  of  succeeding. 

(9)  There  are  agency  officials  who  would  like  to  get  OR  started,  and  would  like  some  impartial 
advice,  but  who  aren't  quite  sure  how  to  get  it.  Furthermore,  they  would  like  to  see  at  first  hand  what 
the  nature  of  the  projects  suitable  for  such  a  group  might  be. 

(10)  The  agency  OR  groups  need  to  have  an  inventory  of  external  resources  available  to  help 
him  in  his  problem  solving:  universities  that  want  graduate  thesis  topics  and  support,  professors  who 
want  a  sabbatical,  colleges  that  are  willing  to  carry  out  a  class  project  on  a  systems  problem  for  an 
agency,  etc.;  contractors  and  their  strong  points  and  experience;  expert  personnel  resources;  data  sources 
that  could  be  relevant  and  could  be  used;  etc.  We  recently  compiled  a  list  of  contracting  agencies  with 
a  Washington  office  and  which  advertised  OR  or  systems  analysis  as  a  specialty,  and  ended  up  with  40 
on  the  first  trial.  RFP  lists  should  be  complete  and  relevant,  and  there  is  no  good  way  of  being  reason- 
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ably  sure  of  this  at  present.  It  is  not  common  knowledge  how  much  or  on  what  basis  one  agency  can  call 
on  another  (e.g.,  Bureau  of  the  Budget  or  Bureau  of  the  Census)  for  substantive  help  or  data  exchange. 

(11)  Currently  employed  methods  of  quality  control  on  OR  studies  could  be  improved  if  experts 
in  other  agencies  were  known  and  could  be  asked  to  participate  in  project  design  or  review. 

(12)  There  is  a  need  for  an  experimentation  laboratory  in  systems  analysis  in  addition  to  the  specific 
agency  opportunities.  Field  experimentation  has  always  been  part  of  a  complete  analysis,  yet  for 
most  purposes,  a  truly  experimental  opportunity  does  not  exist. 

(13)  There  will  soon  be  a  need  for  some  analytic  scheme  to  tackle  the  inter-agency  program 
balance  problem,  be  it  a  system  of  rebuttal  and  debate  with  common  reference  terms,  or  a  government 
management  game,  or  an  inter-agency  analysis  group,  or  some  other  method. 

(14)  There  is  a  need  for  an  OR  textbook  suitable  for  Government  agency  use. 

(15)  There  may  be  a  need  for  an  explicit  mechanism  to  rotate  OR  personnel  among  agencies,  to 
universities  and  back,  etc. 

(16)  There  is  a  need  for  a  census  of  resources   including  hardware,  software,  programs,  computers, 

etc. 

The  recent  rapid  growth  in  operations  research/systems  analysis/multi  disciplinary  problem-solving 
teams  in  the  non-defense  agencies  of  the  Federal  Government  has  opened  a  challenge  of  unusual  propor- 
tions to  the  O.R.  community. 

At  issue  are  questions  concerning  the  development  of  analytic  methods,  test  methods,  data  systems, 
and  means  of  drawing  inferences  in  studies  whose  purpose  is  to  improve  the  operations  of  the  non- 
defense  agencies  of  government.  Questions  related  to  the  definition,  the  criteria,  the  measures,  and  the 
means  of  measuring  the  effectiveness  of  non-defense  programs  are  but  the  beginnings  of  the  work  ahead. 
There  is  the  problem  of  determining  the  best  program  mix  within  an  agency  whose  missions  are  stated 
in  many  different  ways  and  which  sometimes  appear  irreducible  to  a  single  common  scalar  purpose.  There 
is  the  question  of  how  to  determine  the  effectiveness  of  a  Federal  program  when  it  is  designed  to  assist 
in  improving  individuals,  communities,  states,  and  regions,  especially  since  the  instruments  of  bringing 
about  that  improvement  are  in  the  hands  of  many  sources  of  power  in  the  democratic  system  of  rebuttal 
and  debate  that  characterizes  the  American  society. 

Accordingly,  operations  research  in  the  non-defense  agencies  is  confronted  with  the  unavoidable 
task  of  discussing  issues  that  are  at  the  heart  of  our  democratic  society.  It  is  important  that  analytical, 
methodological,  and  substantive  discoveries  made  in  one  agency  that  are  suitable  for  use  by  another 
agency  be  made  a  portion  of  the  Government  memory  bank  as  easily  and  quickly  as  possible. 

One  of  the  aftermaths  of  the  O.R.  Conference  was  an  expressed  need,  as  articulated  by  many  groups, 
for  the  following  two  kinds  of  activity  as  a  minimum. 

(1)  The  convening  of  small  informal  ad  hoc  discussion  sessions  by  agencies  of  the  Government  to 
which  the  O.R.  people  from  other  agencies  are  invited. 

(2)  The  maintenance  by  some  agency  of  an  up-to-date  mailing  list  of  all  those  government  agencies 
O.R.  groups,  universities,  and  contractors,  who  should  be  informed  of  relevant  items,  including  an  up-to- 
date  inventory  of  on-going  O.R,  projects  and  people  finders  in  Government. 

It  appeared  that  each  agency  might  want  to  take  its  own  initiative  in  convening  informal  group 
meetings  of  interest  to  itself.  The  Technical  Analysis  Division  of  the  National  Bureau  of  Standards 
volunteered  to  perform  the  second  function  insofar  as  it  was  able  to  do  so.  In  the  course  of  preparing 
for  the  symposium,  we  had  already  circulated  questionnaires  inquiring  as  to  O.R.  activities  in  various 
agencies.  The  wide  variations  in  respondents'  interpretations  of  "operations  research"  precluded  the 
replies'  being  sufficiently  consistent  to  warrant  inclusion  in  these  Proceedings,  but  the  experience  gained 
should  prove  valuable  to  us  in  designing  a  more  carefully  phrased  survey. 

Prior  to  the  Conference,  a  number  of  agencies  mailed  us  summary  statements  about  the  contents 
of  their  operations  research  programs.  These  were  posted  at  the  entrance  to  the  auditorium  at  the  time 
of  the  meeting.  These  forms  appear  to  be  a  good  starting  point  for  the  creation  of  an  informal  and 
frequently  updated  notebook  of  agency  O.R.  work  that  might  be  circulated  widely,  both  among  the  agen- 
cies and  to  external  institutions  such  as  consulting  firms  and  universities  which  might  assist  in  solving 
agency  problems.  Our  current  plan  is  to  issue  that  first  set  of  reports  as  a  separate  document,  and 
then  to  undertake  the  administrative  chore  of  issuing  updated  versions  of  it. 

We  deeply  appreciate  your  participation  in  this  conference  and  sincerely  hope  that  this  volume  will 
serve  a  continuing  useful  purpose  for  you.  We  hope  that  it  will  be  but  the  first  of  a  long  series. 

W.  E.  Cushen,  Chief 
Technical  Analysis  Division 
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Director 
National  Bureau  of  Standards 
Washington,  D.  C.  20234 


The  National  Bureau  of  Standards  has  a 
statutory  responsibility  to  provide  scientific 
and  technical  advice  and  assistance  to  other 
Federal  Government  agencies.  It  is  in  this 
capacity  that  the  National  Bureau  of  Stand- 
ards has  sought  to  apply  operations  research 
to  the  solution  of  a  variety  of  technical  prob- 
lems of  the  Federal  Government  and  has  served 
as  the  sponsor  for  this  conference.  In  seeking 
to  develop  techniques  and  opportunities  for  the 
wider  application  of  operations  research  in  the 
Federal  Government,  we  wish  to  pay  tribute 
to  the  Department  of  Defense  for  the  leader- 
ship it  has  taken  in  this  area  for  more  than 
two  decades.  Essentially  we  are  seeking  to 
achieve  throughout  the  Government  establish- 
ment the  type  of  benefits  whose  possibility 
has  already  been  demonstrated  by  the  Defense 
Agencies. 

The  National  Bureau  of  Standards  has  still 
another  interest  in  the  development  and  exten- 
sion of  operations  research.  The  central  con- 
cern of  the  NBS  is  in  advancing  and  utilizing 
the  science  of  physical  measurement.  We  be- 
lieve, as  did  Lord  Kelvin,  that  when  you  can 
measure  what  you  are  talking  about  and  ex- 


press it  in  numbers  your  knowledge  is  much 
more  useful  than  when  you  cannot  measure  it. 
In  operations  research  we  are  seeking  to  ex- 
tend the  measurement  process  into  areas  where 
subjective  judgment  rather  than  measurement 
has  been  traditionally  used.  Our  interest  in 
operations  research  goes  back  many  years.  The 
Operations  Research  Society  of  America  grew 
out  of  a  symposium  held  at  NBS  under  the 
auspices  of  our  Applied  Mathematics  Division. 
Several  years  ago  we  established  in  that  Divi- 
sion a  section  for  operations  research,  and  a 
year  and  a  half  ago  we  established  a  Technical 
Analysis  Division  to  serve  as  a  center  for  the 
application  of  operations  analysis  to  the  evalua- 
tion of  problems  within  the  Department  of 
Commerce  and  in  other  interested  Government 
agencies. 

With  this  background  information  you  will 
readily  understand  my  pleasure  in  welcoming 
you  to  this  symposium.  I  trust  it  will  prove 
fruitful  in  advancing  both  practical  and  meth- 
odological aspects  of  the  application  of  opera- 
tions research  to  the  many  pressing  and  diffi- 
cult problems  confronting  Government  agen- 
cies today. 
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Operations  Research  in  the  Civilian  Sector  of  the  Government 


Professor  Merrill  M.  Flood 

University  of  Michigan 
Ann  Arbor,  Michigan  481004 


It  is  a  rare  privilege  for  me  to  meet  with 
you  on  this  occasion,  and  to  have  an  oppor- 
tunity to  express  some  of  my  views  regarding 
the  great  opportunities  for  the  future  of  opera- 
tions research  in  the  civilian  sector  of  the  gov- 
ernment. Although  I  speak  often,  and  freely, 
about  operations  research  opportunities  with 
respect  to  major  national  problems,  and  es- 
pecially in  connection  with  those  not  directly 
related  to  national  defense,  it  is  rare  that  I 
have  an  opportunity  to  discuss  such  matters 
with  those  of  you  actually  responsible  for  the 
work  in  the  federal  government. 

I  believe,  quite  sincerely,  that  you  have  an 
unparalleled  opportunity  and  one  unique  dur- 
ing our  time  for  any  nation  in  the  world,  to 
demonstrate  that  the  approach  of  the  scientist, 
the  engineer,  the  mathematician,  the  econ- 
omist, the  behavioral  scientist,  and  others 
knowledgeable  about  the  scientific  method,  and 
adept  in  its  use,  can  indeed  contribute  substan- 
tially in  the  future  to  the  solution  of  some  of 
our  most  important  problems  at  the  national 
level. 

I  would  not  wish  to  take  the  time,  nor  is  it 
at  all  necessary  with  this  distinguished  group  of 
operations  researchers  and  systems  analysts, 
to  attempt  one  more  definition  of  "Operations 
Research,"  or  to  offer  one  more  listing  of  cur- 
rently relevant  techniques,  or  to  give  one  more 
cataloging  of  the  important  tools  used  in  our 
trade.  Suffice  it  to  say  that  I  share  with  all  of 
you  the  belief  that  our  kit  of  techniques  and 
tools  gets  better  each  year,  and  that  our  shared 
experiences  in  their  use  makes  us  increasingly 
skillful  and  therefore  more  successful  as  the 
years  go  by. 

Nevertheless,  I  also  feel  sure  that  you  will  all 
agree  that  we  have  had  very  modest  success 
indeed,  in  comparison  with  our  expectations 
for  the  future ;  or  as  judged  objectively  by  our- 
selves and  others  as  we  assess  past  accomplish- 
ments, both  inside  the  federal  government  and 
elsewhere.  I  hasten  to  add  that  this  is  not  in  any 
way  intended  to  minimize  the  very  substantial 
progress  already  made  with  respect  to  national 
defense — in  this  case  even  at  the  highest  policy 
levels.  There  has  certainly  also  been  progress 
outside  the  Federal  government,  but  as  yet  pri- 
marily at  lower  operating  levels.  I  am  sure  that 
we  are  all  vastly  impressed  with  the  successes 
of  Secretary  McNamara,  Mr.  Hitch,  Dr.  Entho- 
ven,  and  their  colleagues,  in  introducing  sys- 


tems analysis  and  other  management  and  plan- 
ning methods  into  the  Defense  Department  so 
effectively  during  recent  years.  I  am  sure  that 
we  all  share  the  feeling  that  President  Johnson 
has  given  this  approach  a  tremendous  boost  by 
his  endorsement  with  respect  to  other  depart- 
ments of  the  Federal  government;  I  look  for- 
ward to  hearing  more  about  this  from  Mr. 
Haldi,  your  next  speaker. 

You  may  or  may  not  be  disappointed  to  learn 
that  I  do  not  intend  here  to  suggest  openings 
for  future  effective  application  of  OR  skills, 
primarily  because  my  time  is  too  limited  for 
this.  Instead  I  shall  concentrate  upon  one  im- 
portant national  and  world  development  that  I 
feel  is  particularly  significant  for  the  immedi- 
ate future,  in  the  context  of  the  discussions  in 
this  conference. 

.  My  remarks  will  concentrate  upon  opportu- 
nities for  communication  among  all  of  you  who 
are  concerned  with  the  application  of  O.R. 
skills,  for  I  believe  that  the  civilian  sector  of 
the  Federal  government  has  a  special  charac- 
teristic that  makes  this  opportunity  exception- 
ally attractive  in  comparison  with  similar  work 
for  national  defense  or  in  competitive  industry. 
I  refer  to  the  fact  that  the  detailed  results  of 
investigations  can  normally  be  passed  freely 
among  all  workers  both  during  the  progress 
of  the  studies  and  when  they  are  completed — 
and  perhaps  even  more  important,  after  there 
is  an  opportunity  to  gauge  the  degree  of  suc- 
cess, or  lack  of  success. 

You  are  all  familiar  with  the  considerable 
difficulty  encountered  in  communicating  about 
studies  connected  with  defense  problems.  In 
spite  of  the  many  efforts  made  to  bring  defense 
operations  researchers  together,  often  under 
the  barrier  of  security  classification,  there  has 
been  relatively  little  communication — at  least 
in  the  sense  of  free  and  open  publication  and 
discussion  of  the  type  that  has  seemed  to  be  so 
vital  for  progress  in  the  world  community  of 
scientists.  This  conference  is  an  example  of  the 
kind  of  interchange  that  is  possible,  and  should 
be  encouraged  and  promoted  increasingly  in 
the  future. 

However,  I  am  stressing  quite  another  type 
of  communication,  or  of  information  exchange 
as  I  should  perhaps  call  it,  that  I  believe  will  be 
increasingly  important  and  especially  signifi- 
cant for  O.R.  work  such  as  yours.  We  seem  to 
be  on  the  threshold,  finally,  of  the  next  great 
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advance  in  our  system  for  storing  and  com- 
municating information  of  all  kinds.  I  refer,  of 
course,  to  all  those  kinds  of  things  that  we 
would  bring  into  this  conversation  if  we  were 
talking  about  computerized  information  nets, 
automated  libraries,  electronic  data  centers, 
and  so  forth.  And,  of  course,  I  include  in  this 
development  ordinary  communications  as  rep- 
resented by  publication,  both  in  the  traditional 
sense  and,  in  the  future,  in  new  ways  through 
the  information  transmission  net.  These  devel- 
opments may  even  affect  the  nature  of  our  con- 
ferences and  meetings  in  the  future,  if  and  when 
we  can  afford  to  interact  frequently  and  effec- 
tively over  a  communications  system  and  with- 
out need  to  gather  together  physically  for  such 
interchanges. 

I  shall  narrow  my  remarks  even  further,  to 
a  very  special  aspect  of  this  newly  developing 
national  and  international  information  net- 
work. I  refer  to  the  storage  and  transmission 
of  raw  data,  or  summaries  of  data,  or  formal 
models  that  represent  large  bodies  of  data  com- 
pactly, and  all  other  "facts"  that  can  be  stored 
and  made  accessible  quickly  and  efficiently 
through  such  information  network  facilities. 

Important  among  these  many  forms  of  stored 
models  are  the  increasingly  powerful  computer 
programs  that  we  can  all  share  in  use  while 
profiting  from  the  systematic  synthesis  they 
provide,  often  as  a  result  of  the  work  of  a  quite 
small  group  of  persons. 

More  specifically,  I  propose  as  well  worth 
your  careful  consideration,  the  possibility  that 
all  operations  researchers  throughout  the  fed- 
eral government  can  look  toward  the  develop- 
ment of  great  information  files  that  will  be 
accessible  for  all  who  have  a  real  need.  This 
constitutes  a  very  important  new  form  of  com- 
munication, since  the  information  stored  by  one 
group  need  be  made  available  to  others' only 
after  they  recognize  a  real  need  for  it,  and 
then  in  a  form  where  it  has  been  precisely  de- 
scribed and  is  easily  used. 

Some  may  say  that  my  proposal  is  equally 
applicable  to  many  other  kinds  of  persons  and 
activities.  Although  this  is  true  to  some  extent, 
and  is  certainly  true  very  importantly  in  some 
instances,  I  am  arguing  that  the  analysts 
throughout  the  government  are  in  the  best  posi- 
tion to  develop  these  information  stores  so  that 
they  are  indeed  reliable  and  can  indeed  be  used 
easily.  Nor  do  I  mean  to  limit  this  to  relatively 
simple  tasks  such  as  the  standardization  of 
accounting  data,  or  programming  and  budget- 
ing data,  important  as  these  are;  rather,  the 
intent  is  to  include  as  data  such  things  as  a 
highly  perfected  and  validated  simulation  of  a 
major  agency  or  operation  within  the  govern- 
ment. I  hasten  to  remark  for  the  benefit  of  any 
who  may  feel  that  I  am  far  too  optimistic  about 
such  hopes,  that  I  would  expect  this  to  be  a 


gradual  development  but  will  be  very  surprised 
if  it  is  not  clearly  significant  within  ten  or 
fifteen  years  from  now. 

Let  us  consider  the  proposal  from  a  rather 
different  point  of  view,  stressing  the  impor- 
tance of  working  toward  a  systematic  under- 
standing through  modelling,  of  larger  and 
larger  portions  of  the  Federal  government.  It  is 
probably  impractical  currently  to  construct  a 
valid  mathematical  model,  or  useful  computer 
simulation,  of  the  entire  Department  of  State  i 
or  Department  of  Justice,  or  even  of  major 
subordinate  bureaus  or  agencies.  But  I  would 
not  consider  it  naive,  or  undesirable,  to  begin 
the  attempt  immediately. 

I  am  reminded  of  the  story  about  President 
Jefferson  when  he  was  impatient  with  his  gar-  j 
dener  about  planting  a  tree  and  immediately 
responded,  when  reminded  that  the  tree  would 
not  reach  maturity  for  20  or  30  years,  that  this 
made  it  all  the  more  imperative  to  plant  it 
that  same  day.  It  is  in  this  spirit  that  I  urge  | 
the  importance  of  keeping  this  larger  goal  in  1 
mind  over  the  coming  years,  as  you  all  meet 
with  success  in  analyzing  and  understanding 
larger  and  larger  segments  of  Federal  govern- 
mental activities. 

For  example,  and  from  a  purely  practical 
standpoint,  I  would  think  it  not  too  soon  now  to 
begin  the  development  of  models  that  might 
eventually  enable  us  to  evaluate  tradeoffs  be- 
tween such  major  activities  as  education  and 
justice.  More  specifically,  it  should  be  very 
instructive  and  helpful  to  make  a  study  that 
attempts  to  evaluate  the  tradeoffs  between  an- 
other million  dollars  for  the  FBI  in  contrast  to 
another  million  dollars  for  educational  research, 
or  for  care  of  the  aged,  or  for  foreign  aid,  or 
for  any  one  of  the  other  major  national  pro- 
grams that  surely  are  all  directed  toward  the 
same  set  of  national  goals.  This  is  not  at  all  to 
say  that  such  work  will  be  easy,  or  yield  imme- 
diate results,  but  it  will  happen  sooner  and 
better  if  an  early  attempt  is  made  to  develop 
the  study  activities  in  a  way  that  will  increase 
the  likelihood  of  eventual  success. 

How  might  some  relative  simple  steps  be 
taken  now  to  promote  progress  of  the  kind  I 
have  proposed?  Obviously  there  are  many  pres- 
ent activities  that  are  substantial  and  already 
making  contributions  in  this  sense.  In  par- 
ticular, the  activity  that  Mr.  Haldi  represents, 
will  surely  yield  very  important  ingredients  in- 
deed for  just  such  tradeoff  evaluations. 

My  primary  suggestion  is  that  an  explicit 
attempt  be  organized  and  made  to  accelerate 
this  advance  by  cooperation  among  all  of  you 
in  the  development  of  a  computerized  informa- 
tion network  that  will  store  such  information 
and  make  it  much  more  readily  accessible 
throughout  the  Federal  government  for  use  by 


operations  researchers,  and  for  others  having 
need  for  such  a  resource. 

As  J.C.R.  Licklider  often  puts  it,  you  can 
create  your  own  online  intellectual  community. 

My  own  first  association  with  an  operations 
research  type  of  effort  within  the  Federal  gov- 
ernment was  a  little  over  30  years  ago  when 
I  worked  with  a  distinguished  panel  of  tech- 
nical experts  on  an  advisory  task  for  Attorney 
General  Homer  Cummings.  Our  panel  was  part 
of  a  broad  study  under  Dean  Wayne  Morse, 
now  Senator  Wayne  Morse,  that  was  intended 
to  bring  expert  advice  to  the  Attorney  General 
in  connection  with  his  broad  responsibility  for 
parole  and  release  procedures.  This  panel  was 
chaired  by  the  late  Professor  Samuel  Stouffer, 
eminent  sociologist,  and  included  among  others, 
Samuel  Wilks  and  Frederick  Stephan,  both  dis- 
tinguished statisticians.  The  panel  also  included 
criminologists,  economists,  psychologists,  math- 
ematicians, and  lawyers  and  was  supported  by 
a  strong  staff  group. 

We  could  barely  discuss  basic  questions  such 
as  the  negative  social  value  of  a  released  mur- 
derer, or  other  criminal,  as  compared  with  the 
social  value  of  keeping  him  in  custody  or  under 
parole  supervision.  It  was  my  impression  then, 
at  a  time  when  I  was  engaged  in  studies  of  this 
kind  in  New  Jersey,  that  we  knew  essentially 
nothing  about  major  social  questions  of  this 
type,  nor  did  we  have  the  talents  or  the  trained 
personnel  to  tackle  them.  From  what  little  I 
know  at  present  about  this  field,  I  suspect  that 
we  remain  pretty  much  in  that  same  position 
today,  except  for  the  fact  that  we  certainly  have 
new  talents  and  much  more  expert  personnel 
who  could  tackle  such  problems — perhaps  with 
considerable  success. 

The  newly  established  National  Commission 
on  Crime,  and  similar  commissions  that  have 
been  established  recently  in  some  of  our  states, 
reflect  a  growing  public  awareness  of  the  im- 
portance of  dealing  with  major  social  problems 
more  systematically  and  more  effectively. 

My  most  recent  operations  research  kind  of 
service  for  the  federal  government  was  a  study 
that  I  completed  this  past  September  for  the 
National  Commission  on  Technology,  Automa- 
tion and  Economic  Progress.  My  assignment 
was  a  study  of  the  national  information  net- 
work development,  or  the  "computer  utility" 
as  it  is  sometimes  called  at  MIT  by  Professor 
Fano  and  others.  My  report  is  just  now  being 
published  by  the  National  Commission,  but 
Dr.  Cushen  has  asked  me  to  comment  regarding 
some  of  its  conclusions  —  I  am,  of  course, 
pleased  to  have  this  chance  to  do  so. 

Briefly,  I  am  impressed  by  the  great  oppor- 
tunity afforded  our  nation  in  terms  of  the 
rapidly  developing  information  network,  which 
is  a  good  and  exciting  prospect  indeed,  but  I  am 
disappointed  by  the  slow  pace  at  which  this 


will  all  move  unless  steps  are  taken  by  the 
Federal  government  and  industry  to  anticipate 
and  correct  some  of  the  glaring  deficiencies  in 
our  communications  system  and  in  our  laws 
and  administrative  practices.  I  suppose  that 
none  of  the  difficulties  I  foresee  has  already 
been  observed  and  discussed  somewhere  within 
government  or  industry,  but  I  concluded  on  the 
basis  of  my  study  that  a  great  deal  could  be 
done  to  improve  the  situation. 

For  example,  the  common  communications 
carriers,  such  as  the  American  Telephone  & 
Telegraph  Company  and  the  Western  Union 
Telegraph  Company,  are  moving  very  vigorous- 
ly to  develop  and  construct  a  national  communi- 
cations network  that  will  permit  the  speedy  and 
efficient  transmission  of  data  as  well  as  voice 
messages.  However,  it  is  generally  agreed  by 
experts  on  communications  systems,  who  are 
also  knowledgeable  about  data  transmission 
systems,  that  this  must  be  a  "store  and  for- 
ward" system,  rather  than  a  system  which 
merely  provides  an  exclusive  line  connecting 
two  users  for  some  period  of  use.  In  other 
words,  data  will  originate  at  many  individual 
nodes  on  the  network,  then  be  transmitted  to 
intermediate  nodes  where  it  will  in  turn  be 
sorted  and  re-routed,  perhaps  after  temporary 
storage,  and  eventually  reach  "terminal" 
nodes,  where  it  will  often  be  processed  and 
sent  back  through  the  nodes  of  the  system, 
often  through  quite  complex  paths. 

All  of  this  requires  new  kinds  of  electronic 
switching  centers,  computerized  to  handle,  store 
and  forward  activity  of  this  type,  and  industry 
is  moving  ahead  to  provide  this  type  of  service. 
Unfortunately,  data  transmission  demand  will 
surely  exceed  transmission  capacity  in  the  very 
near  future.  The  need  is  to  take  steps  soon 
which  will  minimize  this  disparity  between 
capacity  and  demand  in  future  years;  my  re- 
port suggests  a  few  moves  that  might  be  made 
toward  this  goal. 

For  example,  there  seems  now  to  be  no  single 
Federal  agency  that  is  clearly  and  obviously 
charged  with  responsibility  for  insuring  that 
this  important  development  does  take  place 
within  our  nation  at  the  proper  pace. 

Another  example,  and  a  lesson  learned  from 
experience  with  some  of  our  major  time-shared 
computing  systems,  such  as  one  at  MIT  and 
another  at  the  System  Development  Corpora- 
tion, is  the  importance  of  bringing  the  future 
users  into  the  system  early  to  help  ensure  that 
its  developing  direction  is  a  good  one. 

Toward  these  ends,  the  report  proposes  that 
at  least  one  major  national  information  net- 
work be  sponsored  by  the  Federal  government 
in  the  near  future  and  that  several  major  sys- 
tems be  made  available  in  the  immediate  future 
for  use  by  individual  citizens,  as  students,  or 
housewives  or  merchants,  and  so  on.  The  report 
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also  stresses  a  number  of  social  problems  that 
require  changes  in  legislation,  or  in  regulations, 
or  in  administration,  if  we  are  not  to  suffer 
from  too  leisurely  development  of  the  national 
information  network  facility.  The  report  also 
urges  that  the  Federal  government  look  very 
carefully  at  its  own  internal  operations  to  en- 
sure that  this  type  of  network  capability  is 
made  available  and  used  appropriately  soon 
by  Federal  agencies. 

It  is  in  this  spirit,  within  the  Federal  govern- 
ment, that  I  believe  there  is  an  unusual  oppor- 
tunity for  analysts  to  cooperate  in  specifying 
the  nature  of  such  a  system  in  order  that  it 
meet  their  needs  later.  Meanwhile,  there  will  be 
many  opportunities  for  all  of  you  to  exert  your 
influence  through  usual  channels  as  this  devel- 
oping information  network  progresses  within 
the  government,  and  for  your  agencies. 

It  may  strike  you  as  strange  that  I  have 
devoted  most  of  my  time  to  a  proposal  that 
rests  so  heavily  upon  recent  technological  de- 
velopments in  information  networks  within  the 
Federal  government  and  elsewhere.  I  have  done 
this  partly  because  of  my  recent  experiences 
with  such  activities,  but  primarily  because  I 
have  been  deeply  interested  in  these  develop- 
ments for  I  feel  that  they  are  so  centrally 
important   for  the   future  of  scientific  and 


analytical  efforts  to  improve  operations  and 
management  within  the  Federal  government 
and  elsewhere. 

I  close  by  repeating  my  opening  remark,  that  I 
the  special  opportunity  that  seems  to  me  to  be 
available  to  all  of  you  working  within  the  Fed- 
eral government  on  problems  primarily  within 
the  civilian  sector  is  the  relative  openness  of 
this  system  and  the  opportunity  you  have  to 
communicate  with  each  other  regarding  your 
work  both  while  it  is  in  progress  and  after  it  is 
completed.  Furthermore,  I  see  the  developing 
information  networks  providing  you  with  a 
truly  advanced  method  of  communications,  or 
information  transfer,  since  you  can  in  the 
future  share  your  experiences  and  your  data  by 
storing  them  in  the  information  network  and 
taking  advantage  of  that  store  as  you  each  ;i 
proceed  with  your  own  work. 

If  the  information  network  facility  will  be 
with  us  in  the  near  future,  then  I  predict  that  I 
you  will  be  sorely  pressed  to  develop  good  meth-  9 
ods  in  time  for  preparing  your  experiences  for 
storage  in  that  system,  so  that  they  can  in  fact 
be  useful  and  be  used  by  others  working  on 
similar  problems.  Just  as  President  Jefferson 
remarked  with  respect  to  his  tree,  since  it  will 
take  so  long  to  finish,  you  should  begin  im- 
mediately. 

I 
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I  welcome  this  opportunity  to  discuss  the 
Planning-Programming-Budgeting  (PPB)  sys- 
tem with  you.  Historically,  suggestions  for  Fed- 
eral program  budgeting  date  from  1912,  when 
a  Commission  reporting  to  President  Taft  pro- 
posed that  a  "budget  bureau"  be  established 
and  that  Government  agencies  present  their 
budgets  along  program  lines.  More  recently, 
two  Hoover  Commissions  have  recommended 
the  improvement  of  Federal  management 
through  the  adoption  of  program  budgeting. 
Before  Mr.  McNamara  came  to  the  Depart- 
ment of  Defense,  many  agencies  already  had 
some  form  of  this  type  of  budgeting.  Many 
facets  of  PPB  are  in  use  in  certain  Govern- 
ment agencies. 

It  is  probably  the  combination  of  Planning, 
Programming,  and  Budgeting  into  an  integrat- 
ed system  which  is  most  new  to  all  Government 
agencies.  The  system,  which  most  economists 
view  as  a  straightforward  application  of  com- 
mon sense,  has  a  few  fundamental  concepts. 
All  agencies  are  being  required  to:  (1)  define 
their  activities  by  objective  within  what  is 
called  a  program  structure,  (2)  prepare  a  mul- 
ti-year program  and  financial  plan,  (3)  trans- 
late program  decisions  into  the  conventional 
appropriation  format,  and  (4)  provide  for 
periodic  reappraisals  of  their  major  programs 
and  objectives. 

The  first  step  in  the  installation  of  a  PPB 
system  is  definition  of  outputs  and  drawing  up 
program  structures  or  outlines  grounded  on 
those  outputs.  For  the  last  several  months  all 
major  agencies  have  been  developing  program 
structures  focused  on  what  the  agency  pro- 
duces, whether  for  the  benefit  of  the  public  or 
of  other  Government  agencies.  The  program 
structures  include  categories,  subcategories, 
and  program  elements.  Program  elements  are 
the  smallest  units  in  the  program  structure  of 
an  agency. 

The  major  purpose  of  the  program  structure 
is  to  serve  as  a  tool  for  top  management  review. 
We  hope  that  it  will  lead  managers  to  ask  the 
"right  questions" — as  President  Johnson  has 
said,  to  reexamine  the  programs  to  see  if  we 
are  really  doing  the  right  thing,  or  to  see 
whether  we  are  actually  funding  obsolete  pro- 
grams. The  primary  focus  of  this  exercise  is  to 


make  sure  that  we  are  selecting  the  right  alter- 
native among  plausible  policies,  and  directing 
resources  to  their  best  use.  A  program  struc- 
ture, intended  primarily  for  consideration  by 
the  Secretary,  the  Director  of  the  Budget  Bu- 
reau, and  the  President,  must  be  aggregated 
into  a  few  meaningful  program  categories. 
Here  are  illustrations  of  program  categories. 

Example  1:  A  program  category  in  the  De- 
partment of  Defense  is  "Strategic  Retaliatory 
Forces."  This  program  consumes  billions  of 
dollars.  Subcategories  within  it  are  "Offensive 
Forces"  and  "Defensive  Forces,"  and  Offensive 
Forces  are  broken  down  into  sub-subcategories, 
such  as  "Manned  Bomber  Systems"  and  "Mis- 
sile Systems."  Program  elements  in  the  Defense 
structure  are  B-52's  and  Polar  Missiles.  In  the 
Defense  Department  structure,  the  element 
"Polaris  Missile  System"  is  the  lowest  level  of 
detail  shown.  Yet  this  element  has  a  budget 
which  is  bigger  than  many  total  agency  budg- 
ets. Included  in  the  Polaris  Missile  System  ele- 
ment are  not  only  the  missiles,  but  also  the 
submarines  needed  to  keep  these  missiles  on 
station,  the  men  that  man  the  submarines,  and 
the  tenders  and  other  supply  facilities  neces- 
sary to  keep  these  submarines  operating.  Thus 
the  program  element  is  an  aggregate  of  many 
things.  This  illustrates  the  total  system  con- 
cept. It  enables  management  to  see  at  a  glance 
how  much  of  its  resources  are  devoted  to  this 
specific  element.  The  program  element  indicates 
the  purpose  for  which  these  resources  are 
brought  together,  and  it  provides  a  focus  for 
the  measure  of  performance.  Management's  at- 
tention is  thus  focused  on  the  performance  of 
the  entire  weapons  system  and  not  on  the  per- 
formance of  support  items. 

Example  2:  In  many  instances,  Government 
agencies  do  not  produce  tangible  products.  The 
Department  of  Justice,  for  example,  produces 
no  hardware.  Yet  here,  again,  program  budget- 
ing means  deciding  what  the  money  should  be 
spent  for.  That  is,  the  purposes  for  which  the 
program  is  engaged,  and  what  the  program  ac- 
complishes. The  Department  of  Justice  pro- 
gram structure  would  include  programs  dealing 
with  anti-trust  declaratory  policy,  investiga- 
tions and  prosecutions  under  O.R.  relating  to 
other  laws  which  are  the  direct  responsibility 
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of  Justice,  investigations  for  other  Government 
agencies,  etc. 

Example  3:  The  program  structure  for  the 
Forest  Service  includes  such  major  categories 
as  timber  production,  wildlife,  grazing,  and 
recreation.  Within  the  Department  of  Agricul- 
ture also,  there  are  services  directly  providing 
outdoor  recreation  facilities,  and  also  two  more 
services  indirectly  providing  recreation  through 
conversion  of  crop  land.  This  year,  for  the  first 
time  in  Federal  budgeting,  the  Department  of 
Agriculture  has  pulled  together  all  its  efforts 
on  recreation  and  reviewed  and  analyzed  them 
at  the  Secretary  level.  This  review  is  one  imme- 
diate benefit  of  program  budgeting. 

This  Recreation  example  shows  how  program 
budgeting  can  shed  light  on  Government  pro- 
grams. For  example,  the  Recreational  Advi- 
sory Council  produced  a  study  which  showed 
that  about  80  percent  of  all  Government  ex- 
penditures on  recreational  activities  are  made 
west  of  the  Mississippi  whereas  about  75  per- 
cent of  the  people  live  east  of  the  Mississippi. 
Because  of  PPB,  we  have  now  changed  the 
emphasis  to  providing  more  recreation  facili- 
ties in  the  East. 

Example  W-  Some  agencies,  like  those  just 
discussed,  use  a  mission-oriented  program 
structure.  The  Post  Office  Department,  on  the 
other  hand,  has  a  program  structure  oriented 
toward  function,  because  that  seems  more  ap- 
propriate for  their  needs.  The  four  most  impor- 
tant categories  in  the  Post  Office  program 
structure  are:  (1)  Getting  Mail  into  the  Sys- 
tem, which  consists  of  two  primary  subcate- 
gories— window  operations  and  mail  collection 
services;  (2)  Processing  the  Mail;  (3)  Trans- 
porting the  Mail;  and  (4)  Delivering  the  Mail. 
In  addition,  there  is  the  catch-all  "General  Sup- 
port" category  for  overhead. 

"Processing  the  Mail"  is  an  intermediate 
operation  which  might  appear  to  be  merely  part 
of  a  broader  system.  Why  was  it  made  a  major 
category?  The  reasons  are: 

— Over  $2  billion  is  spent  to  process  the  mail 
and 

— Mail  processing  involves  many  interest- 
ing "trade-off"  possibilities — alternatives 
which  merit  investigation  and  analysis. 

The  dollars  involved  raise  technical  discussions 
all  the  way  to  the  Postmaster  General,  and  mail 
processing  decisions  are  often  made  at  that 
level.  Even  though  mail  processing  is  an  inter- 
mediate output,  there  are  sound,  pragmatic 
reasons  for  considering  it  a  program. 

Although  some  agencies  have  outputs  which 
cannot  be  measured  in  physical  terms,  almost 
all  programs  should  be  output-oriented.  This 
method  shows  what  is  to  be  accomplished,  not 


merely  the  old  accounting  of  object  classes,  or 
inputs  (personnel,  buildings,  vehicles,  etc.) 

Setting  up  a  structure  requires  rigorous  con-  o 
ceptualization,  but  not  many  man-months.  Once 
the  structure  is  set  up,  an  enormous  amount  of 
work  goes  into  the  Programming  task.  This  in-  i 
volves  taking  the  existing  budget  and  convert- 
ing input  categories  into  new  output  format. 
It  produces  what  is  called  the  "crosswalk,"  and  I 
this  is  essential  since  the  traditional  categories 
are  used  for  submissions  to  Congress.  Since 
much  top  management  time  is  devoted  to 
budget  review,  and  since  top  management 
wants  to  think  in  terms  of  programs,  it  is  im- 
portant that  agency  resources  be  structured  in 
this  way.  Programmers  speak  the  planners' 
language,  and  they  are  expert  at  checking  and 
translating  costs.  Through  programming  we 
ensure  that  work  of  the  budget  people  and  the 
planning  people  corresponds  even  though  the 
conventional  budget  structure  is  used  with  the 
planning  structure. 

Once  the  structure  is  set  up,  the  agency  be- 
gins  developing  a  multi-year  Program  and, 
Financial  Plan  (PFP),  which  specifies  the  out- 
puts and  cost  estimates  for  each  program  ele- 
ment. The  PFP  time  span  can  be  as  short  as 
five  years  beyond  the  current  Fiscal  Year  and 
the  next  Fiscal  Year,  or  as  long  as  ten  years 
for  the  Corps  of  Engineers'  water  projects,  or 
even  80-100  years  for  the  Forest  Service  plant- 
ing program.  In  the  PFP  the  agencies  are  asked 
to  reflect  a  comprehensive  plan,  corresponding 
to  the  program  structure.  The  PFP  is  the  inte- 
grated financial  document  for  the  agency.  Three 
sets  of  tables  make  up  the  PFP:  Part  I  sets 
forth  outputs  for  each  year  by  program,  ele- 
ment; Part  II  shows  dollar  costs  associated 
with  each  program  element;  Part  III  shows 
whatever  other  information  the  agency  deter- 
mines is  relevant  and  necessary,  including  al- 
ternative measures  of  outputs.  Agencies  that 
have  not  been  obtaining  such  data  all  along  may 
soon  be  making  somewhat  greater  demands  on 
field  personnel  than  in  the  past. 

Analysis  is  the  cornerstone  of  the  PPB 
system.  We  are  giving  this  major  emphasis —  ? 
forming  planning  and  analytic  staffs  at  the 
Secretary  level  where  they  do  not  exist,  and  f 
improving  those  which  already  exist.  Without  \ 
much  better  analysis  than  we  have  previously  ] 
had,  this  whole  effort  would  fail. 

This  year  the  budget  submission  for  the  g 
spring  preview  will  consist  solely  of  the  Pro- 
gram and  Financial  Plan  (PFP)  and  Program  \ 
Memoranda  (PM)  in  support  of  the  PFP.  The 
Program  Memoranda  are  to  be  hard  analytic 
documents  which  deal  with  the  issues  confront-  ! 
ing  the  agency  and  which  propose  alternatives 
for  study  during  the  year,  before  critical  deci-  i 
sions  are  made.  I  stress  the  word  "alterna-  v 
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lives."  This  is  fundamental  to  the  PPB  system. 
Without  alternatives,  the  analyst  is  likely  to  be- 
come the  decision  maker.  If  a  study  is  presented 
with  costs,  details,  and  recommendations — but 
no  alternatives — then  the  agency  head's  choice 
is  either  to  accept  the  study,  stick  with  the 
status  quo,  or  perhaps,  suffer  a  budget  cut. 
The  reason  a  study  is  usually  commissioned  in 
the  first  place  is  that  the  agency  head  (or  per- 
haps the  President)  has  already  decided  that 
the  status  quo  must  be  changed. 

PPB  does  not  envision  all-powerful  analytic 
staffs.  It  is  envisioned  that  these  staffs  will 
come  up  with  alternatives,  will  study  their  pros 
and  cons,  quantify  their  implications  and  pre- 
sent them  to  the  agency  head.  Under  PPB 
i  the  agency  head  may  become  much  more 
involved  in  decision-making  in  the  future  than 
previously. 

The  Budget  Bureau  is  interested  in  seeing 
how  well  alternatives  have  been  studied.  A  good 


review  of  alternatives  is  fundamental  to  justifi- 
cation of  major  expenditures.  The  Bureau  of 
the  Budget  does  not  and  will  not  make  the 
agency's  decisions.  It  may  make  recommenda- 
tions regarding  programs,  and  it  may  send  pro- 
posals back  for  further  study.  The  Bureau  does 
not  anticipate  doing  studies  for  the  agencies, 
and  it  does  not  intend  to  develop  a  vested  posi- 
tion in  special  analysis.  The  analysis  will  be 
done  in  the  agencies. 

In  the  near- term,  we  hope  to  improve  major 
decision-making.  In  addition,  we  hope  to  see 
improvement  in  daily  management.  If  we  can 
develop  better  measures  of  agency  output,  in 
the  long  run  we  can  use  these  measures  as 
primary  means  of  management  control.  This 
will  also  affect  the  budget  making  process.  In 
the  long  run,  perhaps  in  the  1980's  or  later, 
I  anticipate  that  PPB  will  not  only  give  us 
better  program  planning,  but  also  help  us 
achieve  more  rational  control  of  the  day-to-day 
functions. 


Appendix 


In  accordance  with  the  instructions  in  the 
now  well-known  Bureau  of  the  Budget  Bulletin 
66-3  and  its  Supplement,  Government  agencies 
are  busily  engaged  in  developing  their  Plan- 
ning-Programming-Budgeting  (PPB)  systems. 
The  Program  Evaluation  Staff,  which  I  repre- 
sent, has  been  deeply  involved  in  the  Budget 
Bureau's  recent  task  of  overseeing  the  estab- 
lishment of  the  PPB  system,  in  offering  guid- 
ance, and  in  arranging  for  training  opportuni- 
ties for  the  agency  staffs.  In  this  paper,  I  will 
emphasize  the  "analysis"  and  "planning"  as- 
pects of  the  documentation  and  conclude  with 
some  implications  of  the  advent  of  PPB. 

PROGRAM  STRUCTURES.  Agencies  have 
been  defining  their  activities  in  a  systematic, 
formal  manner  through  categories,  subcate- 
gories, and  program  elements  which  describe 
what  the  agency  does.  For  this  year's  spring 
preview,  that  job  is  now  finished. 

In  some  instances,  however,  the  existing 
structures  still  contain  obvious  shortcomings.  A 
good  program  structure  should  define  agency 
outputs  in  a  way  that  will  be  of  most  use  to 
agency  top  management,  and  some  structures 
can  be  considerably  improved  in  this  regard. 
Future  refinements  must,  therefore,  seek  to  ac- 
complish this. 

A  second  problem  with  some  program  struc- 
tures is  the  lack  of  uniformity  among  agencies 
j  engaged  in  similar  programs.  For  example,  at 
least  three  major  agencies  are  involved  in  pro- 
viding   outdoor    recreational  opportunities: 


Agriculture,  Interior,  and  the  Corps  of  Engi- 
neers. All  three  agencies  now  show  the  recrea- 
tional opportunity  they  provide  by  geographic 
region,  but  the  regions  don't  coincide.  If  each 
of  these  agencies  reported  the  recreational 
opportunities  they  offer  along  similar  geo- 
graphic lines,  it  would  then  be  very  easy  to 
construct  a  total  Federal  recreational  budget 
which  would  show  exactly  (by  area)  what  the 
total  program  amounts  to.  In  many  such  areas 
interagency  uniformity  of  this  type  is  desir- 
able. This  year,  though,  there  simply  has  not 
been  enough  time  for  any  of  this  to  be 
accomplished. 

PROGRAMMING.  This  system  formally  es- 
tablishes the  base  five-year  plan  in  detail,  dic- 
tates how  program  change  proposals  will  be 
processed,  and  determines  when  and  how  the 
base-year  plan  will  be  extended. 

A  better  grasp  of  what  programming  is  about 
can  easily  be  obtained  by  considering  some  of 
the  questions  which  programming  staffs  have 
been  grappling  with  recently.  For  example: 
What  information  should  be  requested  from 
submitting  bureaus  or  agencies?  How  much  de- 
tail should  the  initial  plan  contain?  How  much 
detail  should  the  plan  ultimately  contain?  What 
forms  should  be  used?  To  what  extent  do  De- 
partment of  Defense  forms  and  procedures 
apply  to  an  agency,  and  to  what  extent  is  DOD 
experience  not  applicable? 

The  programming  system  must  provide  for 
a  "crosswalk"  which  will  convert  the  budget  in 
the  new  program  structure  into  the  present 


appropriation  structure.  But  how  much  accu- 
racy and  detail  are  needed  in  this  conversion? 
To  what  extent  is  accounting  precision  neces- 
sary, and  to  what  extent  are  statistical  cost- 
finding  procedures  acceptable?  Should  we  try 
to  allocate  any  joint  costs?  Which  joint  costs? 
What  allocation  procedure  should  be  used?  Or 
should  all  joint  costs  be  put  in  a  "general  sup- 
port" category?  In  passing,  let  me  note  that 
there  are  no  pat  answers  to  these  questions. 
Good  programming  systems  must  be  individual- 
ly tailored  to  the  particular  needs  and  circum- 
stances of  each  agency,  and  good  programming 
systems  therefore  demand  a  lot  of  hard  work. 

Definition  of  the  "costs,"  to  be  shown  in  the 
PFP,  has  been  something  of  a  problem.  Some 
financial  people  have  interpreted  "cost"  with 
much  preciseness.  However,  the  Supplement  to 
Bulletin  66-3  was  written  chiefly  by  economists 
who  were  thinking  of  costs  in  an  analytic  sense 
— that  is,  the  cost  of  the  resources  that  go  into 
the  plan.  Now  it  so  happens  that  one  of  our 
BOB  Circulars  contains  a  precise,  accrued- 
accounting  definition  of  costs  which  includes 
depreciation,  changes  in  inventories,  accrued 
annual  leave  not  taken,  etc.  In  application, 
many  have  interpreted  costs  as  used  in  the 
Supplement  to  Bulletin  66-3  in  terms  of  this 
formal  definition,  while  it  was  our  intention  to 
interpret  costs  variously  for  different  agencies 
in  whatever  way  makes  the  most  sense  from  an 
economic  point  of  view.  For  this  year,  we  must 
live  with  whatever  confusion  we  have  generat- 
ed. In  the  future,  these  multi-year  program 
and  financial  plans  should  be  tailored  to  suit 
the  needs  and  the  requirements  of  the  individ- 
ual programs. 

One  comment  about  costs  which  I  feel  can  be 
made  with  some  degree  of  assurance  is  that  in 
future  years  the  multi-year  plan  will  show  in- 
vestment and  operating  costs  separately  for 
each  program  element.  Many  of  the  problems 
that  we  are  now  trying  to  solve  will  then  be  a 
lot  easier  to  handle. 

MULTI-YEAR  PROGRAM  AND  FINAN- 
CIAL PLAN.  Probably  the  two  most  important 
parts  of  the  whole  PPB  effort  are  output  data 
and  analyses.  These  are  covered  in  two  basic 
documents :  the  multi-year  Program  and  Finan- 
cial Plan  and  the  Program  Memoranda. 

The  Program  and  Financial  Plan  (PFP)  will 
consist  of  three  parts — all  tables.  Part  I  will  be 
a  table  of  outputs,  showing  measures  for  each 
program  element  expressed  as  physical  units 
of  what  an  agency  is  building,  producing,  or 
encouraging,  year  by  year.  Part  II  will  show 
the  financial  implications  of  Part  I;  e.g.,  how 
much  the  units  of  the  building  plan  are  going  to 
cost  each  year.  Part  III  will  contain  relevant 
supplementary  tables. 

Submission  of  a  multi-year  Program  and 


Financial  Plan  to  the  Bureau  of  the  Budget  will 
be  new  to  all  agencies.  The  degree  of  additional 
planning  required  will  vary  considerably  among 
agencies  since  some  have  been  planning  ahead 
only  on  a  year-to-year  basis. 

The  PFP  is  not  intended  to  be  an  analytic 
document  and  it  should  not  be  interpreted  as  a 
refined  cost-benefit  analysis.  We  are  not  going 
to  be  able  to  take  the  cost  in  Part  II,  divide  it 
by  unit  outputs  in  Part  I,  and  automatically 
come  up  with  something  that  says  this  is  the 
"cost  per  unit  of  output"  or  that  the  reverse 
shows  "benefits  per  dollar." 

The  PFP  is  a  summary  planning  document. 
As  a  quantified  expression  of  an  agency's  plans,  j 
it  becomes  a  valuable  document  for  dialog 
among  the  bureau  chief,  the  head  of  the 
agency,  the  Director  of  the  Budget  Bureau,  and 
the  President. 

PROGRAM  MEMORANDA.  This  is  the  j 
analytic  document.  It  should  state  the  basis 
and  objectives  of  the  analysis,  describe  the  con- 
cepts  and  assumptions  that  were  used,  and  con- 
tain  all  the  narrative  associated  with  the  budget 
submission.  Just  as  the  PFP  states  what  is 
planned,  the  Program  Memoranda  explains  I 
why. 

This  year,  of  course,  we  do  not  expect  to  J 
receive  a  thorough  analysis  of  all  programs. 
But  we  do  feel  there  is  time  to  (1)  formulate  I 
reasonably  clear  and  precise  program  objec-  I 
tives,  (2)  define  the  major  issues,  and  (3)  at  ; 
least  mention  some  of  the  principal  alternatives  j 
which  were  weighed  against  the  selections  re- 
flected in  the  FY  1968  budget.  Our  plan  at  BOB 
is  to  give  each  agency  a  critique  on  its  PM's 
and  have  them  reworked  during  the  summer  ! 
and  resubmitted  in  the  fall.  The  final  version  ill 
of  this  year's  PM's  will  be  the  springboard  of 
those  to  be  written  for  the  following  spring  j 
preview.  Through  this  process,  we  hope  to  in-  ,jj 
ject  a  note  of  continuity  to  the  budget  review  j 
process. 

Let  me  elaborate  for  a  moment  on  the  subject  I 
of  alternatives  and  their  role  in  the  budget  j 
review  process.  Past  agency  submittals  to  the 
Budget  Bureau  rarely  suggested  program  alter-  \ 
natives  (except  for  Defense,  which  now  makes  i 
this  a  regular  habit) .  The  fact  that  the  Budget  i 
Bureau  is  now  asking  for  alternatives  to  be  j 
discussed  explicitly  has  caused  many  people  to  i 
ask  me  whether  BOB  intends  to  start  "running  j 
the  aerency"  or  "making  major  agency  deci-  i 
sions."  The  answer  to  such  questions  is  decided-  | 
Iv  NO !  If  you  will  reread  the  instructions  care- 
fully, you  will  note  that  they  do  not  ask  the 
agency  to  submit  alternatives  to  us  for  a  deci- 
sion. Budget  submissions  should  continue  to 
reflect  decisions  made  by  the  agency  head.  How- 
ever, when  significant  sums  of  money  are  in- 
volved, decisions  should  reflect  the  agency's 
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judgment  of  what  constitutes  the  best  choice 
among  the  most  imaginative  alternatives  pos- 
sible. That  is,  major-  decisions  with  large  spend- 
ing implications  should  be  the  consequence  of  a 
systematic  search  process.  (NOTE :  The  Air- 
lift  Sealift  memo  contained  in  the  Supplement 
to  Bulletin  66-3  illustrates  a  proper  procedure 
for  handling  alternatives  in  budget  submittals.) 
A  tenet  of  program  budgeting  is  that  until  the 

;,  best  and  most  efficient  means  of  achieving  a 
stated  goal  has  been  thoroughly  and  systemati- 
cally searched  for  and  analyzed,  major  program 
spending  decisions  are  not  justified.  I  repeat: 
the  Budget  Bureau  is  not  asking  any  agency  to 
abdicate  its  decision-making  responsibility. 
The  Budget  Bureau  is  asking  each  agency  to 
justify  its  decisions  by  showing  that  it  has  dili- 
gently searched  for  the  best  and  most  efficient 
means  of  achieving  stated  goals  and  objectives. 
In  the  past,  justification  of  this  nature  has 
been,  by  and  large,  noticeable  by  its  absence. 
The  old  form  of  "justification,"  which  rarely 

j  contained  the  rational  basis  for  any  justifica- 
tion whatsoever,  is  no  longer  considered 
acceptable. 


ROLE  OF  THE  ANALYST  IN  PPB.  Let  me 
distinguish  between  decision-makers  and  ana- 
lysts in  the  PPB  system.  They  may,  in  rare 
cases,  be  the  same  individuals,  but  there  is  a 
conceptual  difference  between  the  two  roles 
which  is  worth  noting. 

PPB  assumes  that  the  decision-makers  are 
agency  heads  and  their  principal  assistants. 
Analysts  are  people  trained  to  dissect  prob- 
lems, to  look  at  them  different  ways,  to  develop 
alternatives,  to  employ  quantitative  analysis 
where  practicable,  and  to  make  suggestions  for 
the  decision-maker's  consideration. 

The  phrase  "develop  alternatives"  is  really 
the  key  to  the  distinction  between  decision- 
makers and  analysts.  All  too  frequently  a  pro- 
posal submitted  to  the  decision-maker  describes 
but  one  course  of  action.  The  only  choice  the 
decision-maker  then  has  is  to  accept  it  or  reject 
it.  This  procedure  is  neither  logical,  necessary, 
nor  desirable.  It  de  facto  turns  the  analyst  into 
the  decision-maker.  Program  budgeting  strives 
to  separate  the  two  functions  by  having  ana- 
lysts develop  and  present  choices  between 
meaningful  and  imaginative  alternatives. 
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Operations  Research  Research  and  Government  O.R. 


Dr.  Alan  J.  Goldman  2 
National  Bureau  of  Standards 

1.  Introduction 


I'd  like  to  lead  up  to  my  "message  for  today" 
with  a  little  analogy.  It  concerns,  on  the  one 
hand,  the  typical  relationship  between  the  man- 
ager and  the  operations  researcher — and,  on 
the  other  hand,  that  between  everyman  and  his 
physician.  What  I  have  in  mind  is  that,  unfortu- 
nately, most  of  us  don't  visit  a  doctor  unless 
and  until  we're  already  feeling  uncomfortably 
ill — and  when  we  do  arrive,  what  we  long  for  is 
that  famous  "fast,  fast  relief."  Similarly,  an 
operations  analyst's  introduction  to  a  new  task 
is  all  too  often  in  terms  like  these:  "We  have 
this  problem.  It's  hurting  us  badly,  very  badly. 
Please  give  us  a  guaranteed  optimal  solution  by 
yesterday." 

Under  these  circumstances,  it's  pretty  clear 
what  will  happen.  Perhaps,  by  rare  good  for- 
tune, the  problem  will  happen  to  fall  neatly  into 
one  of  those  areas  in  which  both  theory  and 
practice  are  especially  well  developed.  But  most 
of  the  time,  the  "solution"  obtained  will  neces- 
sarily be  of  the  "quick  and  dirty"  variety. 

Such  a  solution  may  (quite  properly)  be 
accepted  and  acted  upon  as  "best  available." 
However,  it  may  in  fact  be  so  crude  as  to  lead 
to  really  unsatisfactory  results — we  all  know 
that  even  our  best  efforts  can't  ensure  the 
soundness  of  an  O.R.  study's  conclusions.  Or, 
the  quick  and  dirty  solution  may  be  technically 
satisfactory,  but  considerably  more  expensive 
to  implement  than  some  alternative  that  was 
missed.  The  proposed  solution  may  even  be 
fairly  acceptable  on  all  counts,  but  the  manager 
will  never  know  how  much  better  an  answer 
might  have  been  forthcoming  if  only  a  little 
more  time  or  effort  could  have  been  spared — 
or,  and  this  is  the  possibility  I  want  to  stress — 
if  only  the  state  of  the  art  had  been  a  little 
more  advanced. 


Revised  text  of  a  presentation  to  the  Operations  Research  Con- 
ference for  Non-Defense  Washington  Area  Government  Agencies. 
4/20/66.  Includes  sections  omitted  from  the  delivered  address  for 
the  sake  of  brevity.  I  am  indebted  to  J.  R.  Rosenblatt  and  J.  M. 
Cameron  for  helpful  comments. 

2  Chief,    Operations  Research   Section,    Applied  Mathematics  Di- 
vision   Institute  for  Basic  Standards,  National   Bureau  of  Stand- 
ards. Washington,  D.  C.  20234.  The  text  is  an  expression  of  the 
1   author  s  views  on   the  subjects  treated,   rather  than   an  "official 
pronouncement"  of  any  sort. 


This  brings  me  close  to  my  message.  If  most 
government  practitioners  of  operations  re- 
search are  feverishly  racing  the  clock  in  at- 
tempts to  solve  pressing  practical  problems, 
then  just  who  is  going  to  advance  the  state  of 
the  art?  There  is  a  quite  obvious  need  for  the 
analog  of  medical  research — for  groups  whose 
dominant  concern  is  not  the  rapid  resolution  of 
specific  agency  problems,  but  rather  the  sys- 
tematic improvement,  extension  and  creation  of 
the  methods  of  operations  research.  This  is 
what  I  like  to  call  "operations  research  re- 
search," i.e.,  research  into  the  techniques  and 
tactics  of  operations  research.  Abbreviation: 
O.R.R. 

A  natural  reaction  is  that  O.R.R.  activities 
are  properly  the  province  of  the  universities. 
And  so  they  are  largely,  but  certainly  not  ex- 
clusively. It's  easy  to  see  why: 

(a)  We're  concerned  here  with  research 
which  is  applied  rather  than  basic,3  i.e.,  takes  its 
main  stimulus  from  present  and  anticipated 
real-world  problems  rather  than  the  general 
urge  to  advance  human  knowledge. 

(b)  More  specifically,  we're  concerned  with 
methods  of  operations  research  needed  in  gov- 
ernment O.R.  studies.  Appraisal  of  these  needs 
requires  extensive  exposure  to  such  government 
studies;  bringing  the  new  methods  into  effec- 
tive use  quickly  also  requires  close  contact  with 
the  broad  stream  of  government  O.R.  activity. 
I  suggest  that  any  university  group  which  finds 
itself  in  such  a  position  needs  to  do  some 
earnest  soul-searching  about  the  primary  func- 
tions of  a  university. 

(c)  Any  university  outfit  will  (and  often 
should)  be  deeply  concerned  with  issues  dis- 
tinctly peripheral  to  the  needs  of  governmental 
O.R.  A  not  infrequent  occurrence  is  the  tenden- 
cy to  slide  the  research  toward  some  area  which 
is  sufficiently  fashionable  academically  as  to 
permit  the  guiding  professors  to  publish  rather 
than  perish,  and  the  guided  graduate  students 
to  achieve  acceptable  theses.4 


3  This  rather  fuzzy  distinction  is  inappropriate  in  many  of  the 
contexts  in  which  it  appears,  but  seems  genuinely  relevant  here. 

1  The  phrase  "not  infrequent"  was  carefully  chosen;  "typical" 
would  have  been  unjust. 
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2.     Government  O.R.R.:  Demand  and  Supply 


For  the  reasons  just  given,  there's  a  clear 
necessity  for  operations  research  research 
groups  within  the  government,  located  at  focal 
points  for  a  number  of  Federal  O.R.  studies. 
Let  me  spell  out  this  necessity  more  clearly. 
Among  the  effective  resources  of  the  operations 
research  practitioner,  I  do  not  count  all  that 
has  been  discovered,  or  even  all  that  has  been 
published,  but  rather  only  those  items  which 
have  been  so  clearly  written  up  and  so  appro- 
priately implemented  (e.g.,  in  a  computer  pro- 
gram) as  to  be  truly  "available"  for  applica- 
tion or  adaptation  within  the  time  and  resource 
limits  of  a  particular  O.R.  study.  On  this  basis, 
as  a  direct  or  consultative  participator  in  a 
number  of  government  studies,  I  can  say  flatly 
(and  with  no  doubt  that  many  of  you  have  had 
similar  experience)  that  the  state  of  the  O.R. 
art  is  all  too  often  a  recognizably  critical  limit- 
ing factor  on  the  quality  of  what  can  be 
achieved  within  an  assigned  deadline  or  budget 
ceiling.  And  instances  of  this  kind  go  unrecog- 
nized in  many  more  cases,  since  we  naturally 
tend  (without  conscious  decision)  to  formulate 
problems  only  in  terms  which  correspond  to  an 
available  "solution"  method. 

What  makes  this  even  more  frustrating  is 
the  presence  of  "observed  regularities,"  i.e., 
the  recognition  of  essentially  the  same  gap  in 
methodology  or  implementation  as  arising  in 
project  after  project,  although  no  one  sponsor 
has  the  time  or  resources  to  permit  attacking 
the  beast  head-on.  So  we  end  up  ad-hoccing  our 
way  through,  in  more  or  less  makeshift  fashion, 
on  each  individual  occasion.  The  efficient  ap- 
proach to  these  recurrent  problems  often  in- 
volves looking  at  a  version  general  enough  to 
refer  not  only  to  Agency  X's  problem  in  1966, 
but  also  to  what  will  be  the  problems  of  ump- 
teen other  agencies  (but  perhaps  not  Agency 
X)  by  1976.  As  we  know,  to  support  such  re- 
search out  of  Agency  X's  1966  budget  is  typi- 
cally unthinkable.  Work  of  this  sort  naturally 
constitutes  the  role  of  an  independently  sup- 
ported operations  research  research  group  with 
the  specific  mission  of  ascertaining  and  re- 
sponding to  these  general  needs  of  govern- 
mental O.R. 

With  the  "demand"  side  of  the  picture  for 
government  operations  research  research  thus 
indicated,  let's  turn  to  the  "supply"  side.  First, 
the  military  O.R.  sector.  Here  one  thinks  at 
once  of  the  outstanding  achievements  of  the 
RAND  Corporation,  then  of  the  many  contribu- 
tions of  ORO/RAC,  OEG/CNA,  WSEG/IDA, 
etc.  (Note,  though,  that  none  of  these  are  exact- 
ly within  government.) 

Now  for  the  civilian  side.  To  exclude  the 
other  "giant"  as  well,  I'll  call  it  the  non-military 


terrestrial  sector,  comprising  government  ac- 
tivities concerned  with  such  minor  items  as  the 
education  and  international  dealings.  Despite 
nation's  economy,  natural  resources,  health, 
some  notable  individual  efforts,  if  one  com- 
pares the  available  resources  for  government 
O.R.R.  in  this  sector  with: 

(a)  the  difficulty  and  number  of  the  prob- 
lems involved,  and 

(b)  the  resources  for  analogous  functions  in 
the  military  sector,  then  it  seems  very  hard  to 
arrive  at  a  rating  significantly  different  from 
zero. 

The  gist  of  my  "message"  is  that  this  just 
won't  do.  It  is  a  platitude  that  yesterday's  O.R. 
methods  will  fall  short  of  adequacy  for  today's 
problems — they  fell  short  for  yesterday's,  and 
today's  are  certainly  no  easier.  Moreover,  I  can  | 
see  a  clear  and  present  danger  that  techniques 
developed  for  military  O.R.  studies  will  be  j 
stretched  well  past  the  validity  point  to  cover 
the  kinds  of  objective  functions  injected  into 
non-military  analyses  by  public  interest  con- 
siderations. 

The  contention  here  is  that  lack  of  apprecia- 
tion for  the  need  for  adequate  civilian  terres- 
trial government  O.R.R.  has  hurt  us  in  the 
past,  and  is  hurting  now.  Without  remedial  ac- 
tion, the  pain  will  rise  to  the  agony  level  as 
PPB  and  its  offspring  make  ever-increasing 
demands  on  the  O.R.  capabilities  of  all  govern- 
ment agencies,  and  thus  on  the  O.R.  art  itself. 
And  these  are  needs  which  can't  be  met  "on  the  j 
cheap,"  in  dedication  or  time  or  dollars. 

Who  should  bell  this  cat  ?  Despite  the  govern- 
ment tradition  that  questions  of  organization 
generally  take  priority  over  questions  of  sub- 
stance, I  have  no  considered  views  to  present. 
In  the  long  run,  it  seems  quite  likely  that  each  l 
Executive  Department  (and  several  of  the  ma-  11 
jor  independent  agencies)  will  require  at  least 
one  O.R.R.  facility  attuned  to  the  peculiarities 
of  its  own  needs.5  In  the  short  term,  it  seems 
apparent  from  my  prejudiced  viewpoint  that  m 
one  natural  locus  for  such  work  will  be  the  i 
National  Bureau  of  Standards,  with  its  tradi-  ll 
tion  of  O.R.-type  services  to  other  agencies,  i 
and  its  special  assigned  responsibilities  regard-  I 
ing  both  (i)  cost-benefit  analyses  and  (ii)  help-  • 
ing  to  achieve  increasingly  efficient  government  1 
use  of  that  very  valuable  O.R.  tool,  the  digital 
computer.  But  my  personal  acquaintance  with  ' 
many  talented  government  O.R.  professionals  J 
just  doesn't  amount  to  the  kind  of  knowledge, 
of  agency  structures  and  traditions,  that  would  i 
permit  me  to  identify  the  other  natural  loci  or  t 
nuclei  which  must  surely  exist. 


5  Such  facilities  would  still  need  access  to  specialized  consultants 
expert  in  the  more  esoteric  spells. 
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3.     Data,  Dissemination,  Documentation 


Before  becoming  a  little  more  specific  about 
promising  areas  for  operations  research  re- 
search, I  want  to  remind  you  of  some  other 
aspects  of  government  O.R.  (or  operations  re- 
search in  general)  which  have  their  own  un- 
pleasant features  to  add  to  the  difficulties  aris- 
ing from  inadequate  methods. 

One  such  aspect  is  the  typical  quality  of  the 
data  with  which  we  must  deal.  It  is  depressing 
to  recall  that  to  the  scientist,  "data"  are  care- 
fully defined  numbers  derived  from  precise 
measurement.53  It  is  even  more  depressing  to 
observe  that  according  to  the  Latin,  "data" 
means  "things  which  are  given,"  i.e.,  are  lying 
at  hand  and  freely  available.  In  contrast,  the 
numerical  material  of  most  O.R.  studies  seem  to 
fall  into  four  categories: 

(a)  Predictions  or  extrapolations  into  the 
future.  My  only  comment  here  is  the  recom- 
mendation that  anyone  who  applies  the  word 
"data"  to  any  glimpse  into  any  crystal  ball 
should  be  shot  as  soon  as  possible  (within 
constraints  of  time  and  budget,  of  course). 

(b)  "Data"  which,  though  essential  to  your 
purpose,  were  so  far  from  "freely  available" 
that  acquiring  them  has  left  you  too  exhausted 
and  broke  to  go  on  and  use  them.  In  the  mili- 
tary O.R.  field,  it  has  not  been  uncommon  for 
inter-service  rivalries  to  force  data-gathering 
to  take  forms  just  short  of  espionage.  I  hope 
this  won't  recur  in  the  civilian  sector. 

(c)  "Data,"  so-called,  which  started  out  as 
reasonably  honest  numbers.  But  they've  been 
aggregated  and  re-aggregated  and  disaggre- 

•  gated  and  interpolated  and  scaled  and  adjusted 
— in  short,  subjected  to  so  many  operations, 
each  somewhat  open  to  question,  that  the  ratio 
of  fact  to  artifact  is  slim  indeed.  You  don't 
want  any  decision  significant  for  our  nation  to 
lean  on  numbers  like  these,  put  together  with 
pipe  cleaners  and  chewing  gum.  You  wouldn't 
even  want  the  fate  of  Northwest  Saratoga  Falls 
to  depend  on  them.  But — they're  what  you've 
got. 

(d)  My  final  category  consists  of  those  num- 
bers which  are  indeed  "freely  available,"  but 
simply  aren't  to  be  believed — they  have  per- 
haps been  prepared  to  bias  the  analysis  you're 
attempting,  or  else  with  no  axe  to  grind  but  in 
total  ignorance  of  proper  sampling  technique, 
or  may  reflect  drastically  different  usage  of  cer- 
tain key  words  by  you  and  by  your  informant. 

This  list  of  horrors  reminds  us  that  O.R. 
studies  often  involve  conceptually  complex 
processing  of  their  input  data;  knowing  how 
far  we  can  go  (and  in  what  directions)  requires 
a  rather  sophisticated  understanding  of  just 


G«  The  contrast  being  drawn  is  too  sharp  to  he  blunted  by  the 
qualifications  needed  to  make  this  over-simplification  respectable. 


what  the  numbers  mean  and  how  they  were 
arrived  at.  It  requires  recognizing  that  the 
"data"  as  received  are  generally  not  raw  strings 
of  digits,  but  rather  have  had  a  structure  im- 
plicitly imposed  on  them  by  the  methods  (ac- 
counting rules,  consistency  criteria,  formulas 
based  on  economic  or  social  or  demographic 
theories)  used  in  checking  and  "adjusting" 
them.  Hopefully  our  models  will  be  compatible 
with  this  structure,  perhaps  even  incorporating 
some  of  its  elements.  Achieving  the  kind  of  un- 
derstanding yearned  for  here,  would  certainly 
be  facilitated  by  close  association  with  the  plan- 
ning and  evolution  of  the  coming  generation  of 
Federal  reporting  systems. 

So  much  for  data.  On  dissemination,  I  simply 
call  to  your  attention  how  much  high-quality 
operations  research  work,  which  could  provide 
essential  starting  points  for  efforts  by  other 
agencies,  is  either  never  written  up  properly, 
or  written  up  only  years  and  years  after  the 
fact,  or  is  written  up  well  and  on  time  only  to 
be  buried  in  the  deepest  dungeon  of  "adminis- 
trative classification"  by  the  agency  involved. 

This  leads  to  a  third  striking  aspect  of  the 
government  O.R.  scene — the  scandalously  low 
average  quality  of  documentation  of  (in-house 
and  contracted)  operations  analyses.  Let  me  de- 
scribe the  standards  used  in  that  appraisal.  In 
looking  at  a  piece  of  O.R.  documentation — from 
the  viewpoint  of  the  analyst,  not  necessarily 
that  of  the  manager  for  whom  the  study  was 
made — what  is  one  justified  in  expecting? 

To  permit  concentration  on  the  intellectual 
content,  the  text  must  be  written  in  clear  and 
reasonably  smooth  style,  without  side  expedi- 
tions to  drag  in  and  display  clusters  of  what- 
ever terms  (e.g.,  "system")  are  believed  to  be 
in  current  fashion.  Turning  to  substance,  we 
note  that  the  motivation  for  reading  the  docu- 
ment is  presumably  a  hope  of  employing  its 
results  as  partial  basis  for  some  decision  or  as 
inputs  to  some  further  analysis,  or  perhaps  of 
adapting  its  methods  for  use  in  some  new  study. 
None  of  these  purposes  is  well-served  unless 
the  serious  reader  can  without  undue  effort 
track  down  what  assumptions  and  approxima- 
tions were  made  (and  with  what  justification), 
what  particular  meanings  were  attached  to  sig- 
nificant, but.  somewhat  ambiguous  terms,  and  so 
on.  Getting  at  this  sort  of  information  is  hard 
enough  today,  when  the  principal  author  is  per- 
haps only  as  far  away  as  the  other  side  of  the 
Federal  Triangle.  And  if  the  present  quality  of 
documentation  remains  typical,  and  if  Profes- 
sor Flood's  predictions  are  realized  and  one  will 
typically  be  applying  X's  model  to  Y's  data 
after  picking  both  up  "blind"  through  com- 
puter tie-in  with  points  1000  miles  distant — 
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then  the  odds  against  producing  anything  but 
garbage  seem  intolerably  high. 

But  there  is  still  another  class  of  readers  to 
consider,  those  anxious  to  find  what  the  course 
of  progress  of  the  documented  study  has  to 
teach  us  in  operations  research.  They  would 
need  to  know  what  alternatives  to  the  approach 
actually  chosen  were  considered,  and  why  they 
were  rejected.  What  was  the  institutional 
framework  in  which  the  work  was  done,  and 
the  backgrounds  of  those  with  whom  and  for 
whom  the  work  was  done,  and  how  did  these 
factors  influence  the  conduct  of  the  study?  If  a 
predictive  model  was  developed,  what  provision 
was  made  for  follow-up  observation  to  permit 
adaptive  corrections,  and  (hopefully  in  a  fol- 
low-up volume)  how  successful  were  the  predic- 
tions and  what  are  the  implications  for  the 
predictive  technique  used? 

Some  of  the  reasons  for  the  present  average 
level  of  O.R.  documentation  lie  fairly  near  the 
surface.  In  the  first  place,  the  kinds  of  material 
demanded  above  are  really  extremely  hard  to 
exposit,  and  can  easily  run  into  agency  clear- 
ance problems.  (Perhaps  this  explains  why  the 
mathematicians — whose  contributions  are  par- 
ticularly easy  to  write  up  and  to  present  ab- 
stractly without  reference  to  the  controversial 
issues  of  a  particular  study — are  now  nearly 
monopolizing  the  O.R.  journals,  thus  reinforc- 
ing the  widespread  impression  that  operations 
research  is  basically  a  collection  of  esoteric 
mathematical  magics.  As  an  operations  re- 
search mathematician,  I  certainly  appreciate 
having  my  colleague's  mathematical  contribu- 
tions to  O.R.  readily  available  to  me  in  print, 
but  I  long  for  a  professional  literature  which 
would  provide  me  with  more  useful  insights  on 
the  all  too  many  facets  of  operations  research 
in  which  my  technical  training  doesn't  leave  me 
fairly  capable  of  fending  for  myself.) 

Second,  project  budgets  almost  invariably 
underestimate,  quite  substantially,  the  drain  on 
time  and  dollars  and  manpower  needed  to  do 
such  recording  right.  This  in  part  reflects  the 
fact  that  much  of  the  quality  described  above  is 
not  strictly  necessary  for  the  short-term  inter- 
ests of  the  agency  funding  the  study.  It  also  re- 
flects the  lack  of  a  tradition,  like  that  in  the 

4.     Communication,  Or 

In  this  and  the  following  section,  I  want  to 
mention  a  few  potentially  important  areas  for 
operations  research  research.  The  list  is  of 
course  quite  an  individual  matter,  reflecting  my 
own  reactions  to  those  difficulties  encountered 
during  the  particular  O.R.  studies  in  which 
I've  happened  to  participate.  Some  of  you 
would  likely  have  selected  a  totally  disjoint  list 
for  special  mention,  but  there  would  probably 
be  substantial  agreement  that  the  subjects 


sciences,  of  documentation  of  a  sort  directed  at 
the  professional  community.  This  lack  is  not 
surprising,  since  so  much  operations  research 
is  carried  out  in  areas  where  considerations  of 
military  security  or  industrial  "proprietary" 
reticence  are  appropriate.  But  these  constraints 
should  not  apply  to  the  civilian  government 
O.R.  sector! 

For  the  agency  sponsoring  an  O.R.  study 
which  has  been  documented  in  the  usual  sloven- 
ly style,  the  consequence  is  that  if  the  manager 
for  whom  the  study  was  done  leaves — and  he 
will — or  if  the  study  team  breaks  up — and  it 
will — then  the  future  potential  return  from  the 
resources  invested  in  the  project  becomes  essen-  » 
tially  nil.  The  benefits  thus  jeopardized,  to  the 
agency  and  to  other  agencies  with  related  prob- 
lems, can  be  quite  substantial. 

For  the  O.R.  community,  the  prevalence  of 
inadequate  documentation  means  systematic 
deprivation  of  precious  opportunities  to  learn 
from  each  other's  insights  and  aberrations. 
Some  of  what  isn't  written  down  does  ulti- 
mately get  communicated  to  a  few  colleagues 
by  word  of  mouth,  but  this  is  the  prehistoric 
mode  of  information  transfer,  scarcely  one  to 
rely  on  for  efficiently  promoting  the  advance  of 
the  state  of  the  art  in  operations  research. 

I  suspect  that  studies  exceeding  a  certain 
(what?)  level  of  effort  should  routinely  include 
a  project  historian — even  though  this  may  rule 
out  a  type  of  doctoral  thesis  (The  Story  of 
Project  Q:  "What  Fools  We  Mortals  Be") 
otherwise  likely  to  be  quite  popular  in  the  next 
decade.  I  hope  to  see  an  increasing  stream  of 
solid  case  histories  published  in  our  journals;  j: 
the  forthcoming  "applications"  issue  of  the 
O.R.  Journal  should  either  set  the  necessary  \ 
precedent,  or  provide  keener  insight  into  the 
difficulties  involved,  or  do  some  of  both.  But 
most  strongly  of  all,  I  want  to  urge  that  in 
allocating  resources  and  energies  among  the  i 
various  phases  of  our  work,  we  remember  that  j| 
in  the  documentation  phase  we  are  not  writing  ; 
exclusively  for  this  decision-maker  and  this 
year,  but  also  for  our  present  and  future  col-  j 
leagues,  and  have  a  heavy  obligation  to  con-  I 
tribute  what  we  can  to  our  profession. 

ganization,  and  Strategy 

sketched  below  do  rank  high  in  significance  and 
promise  for  O.R.R.6 

The  first  topic  is  that  of  communication — 
with  the  manager  or  decision-maker  for  whom  ' 
the  O.R.  study  is  carried  out.  It's  often  said  that 
the  success  of  an  O.R.  project  should  be  judged 


c  In  restrospect,  the  questions  relating  to  the  proper  selection  of 
measures  of  effectiveness  or  benefits  (i.e.,  'objective  functions"), 
and  to  the  related  foundational  field  of  utility  theory,  seem  so 
central  as  to  make  their  absence  from  this  Section  and  the  next 
somewhat  embarrassing. 
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by  the  extent  to  which  its  recommendations  are 
implemented.  I  believe  that's  a  gross  exaggera- 
tion— I  have  to  believe  it,  since  otherwise  I'd 
have  become  so  discouraged  as  to  leave  the  field 
long  ago.  What  is  essential  is  that  the  frame- 
work, the  model  underlying  the  study,  has  been 
communicated  to  the  manager  clearly  enough, 
and  is  sufficiently  convincing  and  illuminating 
to  him,  as  to  substantially  help  him  in  reaching 
a  decision.  If  for  example  his  ultimate  selection 
over-rides  whatever  recommendations  the  O.R. 
report  contains,  then  at  least  he  will  have  a 
good  idea  of  what  has  been  sacrificed,  and  will 
have  been  compelled  to  think  through  (and 
will  therefore  be  prepared  to  explain  to  others) 
what  additional  considerations  compel  this 
sacrifice. 

But  how  often  do  we  achieve  this  degree  of 
communication?  What  can  we  do  during  the 
study  to  increase  its  likelihood?  At  what 
points,  at  which  levels,  and  how  frequently 
should  the  manager  and  his  staff  be  brought 
into  the  study?  A  number  of  articles  have 
offered  sage  (and  sometimes  snide)  counsel  on 
such  topics,  but  I  think  the  time  is  ripe  for 
carefully  designed  experiments,  and  for  in- 
formed empirical  observations,  that  could  shed 
substantial  light  on  these  important  questions. 

Besides  communication  with  the  sponsor, 
there  is  also  the  matter  of  communications 
within  the  O.R.  team  itself.  How  much  in- 
formation should  junior  staff  be  given — with 
the  risk  on  the  one  hand  of  distracting  them 
from  their  assignments,  and  on  the  other  hand 
of  missing  their  potentially  valuable  sugges- 
tions and  failing  to  give  them  a  full  sense  of 
participation?  What  degree  of  consensus 
should  be  required  for  significant  strategy  de- 


5.  Mathematical 

The  topic  of  this  section  is  the  one  closest  to 
my  own  heart.  I  am  a  bit  reluctant  to  display 
my  enthusiasm,  however,  for  fear  of  reinforc- 
ing a  widely  held  image  of  the  mathematician 
in  operations  research  as  infatuated  with  his 
favorite  abstruse  technique  to  the  point  of 
unconcern  with  the  actual  needs  of  the  prob- 
lem at  hand.  This  image  is  in  some  cases  all 
too  well  deserved,  but  I  suspect  that  in  many 
instances  it  arises  from  a  somewhat  over- 
simplified view  of  quantitative  models : 

To  be  at  all  manageable,  a  model  must  omit 
the  overwhelming  preponderance  of  the 
bloomin'  buzzin'  confusion  that  is  reality.  Since 
complete  realism  is  out  of  the  question,  it  fol- 
lows that  there  is  no  one  "right"  model  for  a 
given  situation,  just  more  or  less  suitable 
models.  A  certain  fidelity  to  the  main  features 
of  the  actual  situation  is  obviously  necessary  if 


cisions  in  the  study — e.g.,  does  one  want  to  re- 
ceive unanimous  consent  by  task  leaders,  or 
perhaps  adopt  a  policy  of  "O.K.  if  no-one  ob- 
jects in  writing?"  What's  to  be  done  about  the 
fact  that  the  participants  in  a  5-person  meet- 
ing will  go  off  with  at  least  5  different  impres- 
sions of  what's  been  decided? 

I  have  found  these  questions  to  be  sadly  sig- 
nificant, to  the  point  of  wondering  whether 
each  10  or  so  staff  members  don't  normally 
need  another  man  to  provide  mere  coordination, 
as  distinguished  from  technical  direction.  Here 
again  the  adage  "know  thyself"  seems  appli- 
cable to  O.R. — simulation  and  experimental 
studies  on  these  questions  of  O.R.  project  or- 
ganization seem  well  worth  pursuing. 

There  are  also  some  more  basic  questions  of 
project  strategy.  It  takes  mighty  steady  nerves 
to  "keep  you  cool,"  to  sit  tight  while  systemati- 
cally working  out  a  program  of  activities,  to  do 
preliminary  explorations  of  several  alternative 
approaches,  to  wait  for  clarifying  data  or 
"customer"  resolution  of  ambiguities — all  this 
while  deadline  time  ticks  closer  and  closer. 
There  is  the  compulsion  to  do  something — to 
get  cracking — to  get  something  going  which 
can  be  reported  as  progress  to  the  customer, 
and  most  of  all  to  oneself.  Yet  my  own  experi- 
ence, at  least,  indicates  that  a  deliberate  plan- 
ning phase  is  essential  for  substantial  projects, 
that  15-20%  of  the  allotted  study  period  is 
probably  even  scanty  for  this  phase,  and  that 
shortchanging  it  leads  to  premature  commit- 
ments which  require  agonizing  and  basically 
unsatisfactory  patching  up  toward  the  end  of 
the  job.  Do  we  know  enough,  in  operations  re- 
search, to  be  able  to  study  this  sort  of  decision 
process?  I  would  certainly  like  to  think  so.  Are 
they  worth  studying?  I  have  no  doubt  they  are. 


Methods  and  Models 

relevant  answers  are  to  be  obtained.  But  to 
yield  any  answers  at  all,  the  model  must  be 
tractable  to  whatever  "solution"  or  "manipula- 
tion" operations  are  required.  And  so  a  "best 
fit"  to  the  desired  degree  of  realism  must  be 
chosen  from  among  those  models  for  which 
solution  procedures  do  exist,  not  because  use 
of  these  procedures  is  an  end  in  itself,  but 
simply  because  there's  nothing  else  to  be  done. 

The  role  of  mathematical  O.R.R.  is  to  enlarge 
this  area  of  choice,  of  "solvable"  mathematical 
models,  so  as  to  permit  a  better  fit.  In  cases 
where  no  decent  fit  was  previously  available 
and  one  is  now  provided,  we  find  sudden  recog- 
nition that  instances  of  model  M  are  really 
numerous  and  important — how  could  they  have 
been  overlooked  for  so  long?  Solutions  make 
problems  as  well  as  vice  versa,  i.e.,  it  is  the 
existence  of  an  efficient  solution  method  for  a 
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certain  model  which  creates  the  incentive  to 
investigate  whether  a  situation  under  study 
can't  be  subsumed  under  that  model  without 
unacceptable  distortion. 

There  are  two  areas  of  mathematical  opera- 
tions research  in  which  the  last  few  years  have 
seen  especially  notable  progress,7  with  every 
prospect  of  even  greater  successes  in  the  years 
to  come.  One  is  the  field  of  combinatorial 
optimization,  largely  but  not  exclusively  con- 
cerned with  network  problems  (routing,  sched- 
uling, partitioning,  etc.).  Here  the  "exponential 
explosion" — the  fact  that  the  number  of  alter- 
natives to  be  chosen  among  increases  exponen- 
tially (or  worse!)  with  problem  size — is  being 
coped  with  successfully  in  more  and  more 
cases.  The  second  field  is  that  of  nonlinear  pro- 
gramming, i.e.,  the  maximization  or  minimiza- 
tion of  a  (not  necessarily  linear)  function  of 
many  variables,  perhaps  subject  to  a  number  of 
(not  necessarily  linear)  side  conditions. 

Mathematical  O.R.R.  ought  to  increase  its 
efforts  in  these  fields,  so  as  to  exploit  promptly 
the  present  and  anticipated  breakthroughs. 
Furthermore  the  results  have  to  be  turned  into 
reliable  tools,  readily  available  for  use  during 
O.R.  studies  which  require  them.  This  calls  for 
efficient  well-documented  computer  codes,  as 
well  as  systematic  computational  experiments 
to  compare  alternative  methods  and  learn  which 
among  them  are  best  suited  to  particular  types 
of  problems. 

Of  course  many  of  the  problems  we  will  want 
to  solve  will  still  turn  out  to  be  mathematically 
intractable  with  the  tools  available.  I  would 
hope  to  see  more  emphasis  in  O.R.R.  on  develop- 
ing and  disseminating  rigorous  sophisticated 


procedures  for  approximating  intractable  mod- 
els by  simpler  ones,  in  particular  for  "surround- 
ing" the  original  situation  with  a  number  of 
tractable  ones,  each  deliberately  biased  in  a 
known  way. 

It  is  natural  to  wonder  whether  civilian  gov- 
ernment O.R.  will  have  some  distinctive  de- 
mands to  make  on  mathematics.  One  specula- 
tive notion  is  that  there  will  be  an  increasing 
need  to  determine  appropriate  sorts  of  equili- 
bria rather  than  optima,  that  it  may  be  less 
fruitful  to  seek  a  solution  which  is  "best"  in 
some  sense  than  to  aim  at  striking  a  proper  bal- 
ance between  conflicting  public  interests. 

I  must  confess  to  a  general  uneasiness  about 
the  basic  appropriateness,  for  much  of  opera- 
tions research,  of  traditional  mathematics. 
Mathematics  originated  in  connection  with  hu- 
man activities — surveying,  accounting,  naviga- 
tion— characterized  by  rather  precisely  quan- 
tifiable "data,"  and  a  requirement  for  rather 
precise  numerical  results.  The  further  develop- 
ment of  mathematics  occurred  in  connection 
with  the  sciences  and  engineering,  again  fields 
where  "hard  numbers"  prevail.  It  seems  unlike- 
ly that  techniques  developed  in  such  contexts 
would  be  generically  suitable  for  situations  in- 
volving "data"  scarcely  deserving  of  the  name, 
and  requiring  not  the  precise  evaluation  of 
various  courses  of  action  but  rather  only  their 
ranking  according  to  quite  fuzzy  criteria,  or 
even  just  the  determination  of  the  top  one  or 
two  or  three  among  them.  There  may  well  be  a 
need  for  some  new  kinds  of  mathematical  con- 
cepts and  operations,  or  at  least  a  greater  appli- 
cation in  O.R.  of  the  notions  of  "modern  mathe- 
matics" with  its  stress  on  structures  and  rela- 
tionships rather  than  numbers. 


6.  Summary 


Operations  research  has  been  described  as  an 
art_  which  "gives  bad  answers  to  problems 
which  otherwise  would  receive  even  worse  an- 
swers." This  art  is  being  and  is  going  to  be 
applied,  by  the  non-military  sector  of  our  gov- 
ernment, to  problems  so  important,  so  difficult 
and  so  numerous  as  to  require  every  effort  to 
minimize  the  "badness"  of  our  answers. 

We  can  make  some  progress  toward  this  goal 
by  raising  the  quality  of  our  documentation 
from  its  present  abysmally  low  level,  so  that 
we  can  make  better  use  of  each  other's  work 
and  thus  free  resources  for  resolving  those  diffi- 
culties unique  to  the  problem  at  hand.  We  can 
delve  more  deeply  into  the  reliability  of  the 


7  ^  wo.u]d  be  '"appropriate,  in  a  non-technical  paper  such  as 
this,  to  cite  the  specific  publications  involved. 


"data"  entering  our  work,  and  adjust  our 
methods  and  conclusions  accordingly.  But  even 
with  all  this  done,  we  would  be  faced  with  sub- 
stantial  gaps  between  our  needs  and  the  O.R. 
methods  available  to  us.  Our  work  will  fre- 
quently call  for  knowledge  and  methods  which 
don't  yet  exist,  or  which  exist  in  theory  but  not 
in  practically  usable  form — perhaps  for  mate- 
rial there  was  no  real  incentive  to  develop  in 
the  climate  of  military  or  industrial  O.R.  A 
substantial  in-house  capability  to  advance  the 
art  of  operations  research  for  civilian  terres- 
trial branches  of  government  is  needed,  but 
practically  none  is  not  present.  In  the  present 
paper  I  have  tried  to  indicate  the  reasoning  be- 
hind these  assertions,  and  to  identify  some  of 
the  areas  on  which  a  proper  O.R.R.  effort 
should  focus. 
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Reports  of  Panel  Meetings 

Editor's  Note:  As  noted  in  the  agenda,  three  panel  sessions  were  held  in  the  afternoon:  Panel  I,  Special 
Problems  of  OR  in  Several  Agencies  of  Government;  Panel  II,  Management  Views  in  OR/SA  to  Fulfill 
PPB  Requirements;  Panel  III,  Broadening  the  OR  Competence  of  Mid-Careerists.  The  panel  chairmen 
were  asked  to  prepare  a  set  of  notes  on  the  activities  of  their  panels.  Notes  on  Panel  I  and  Panel  III  were 
received  and  are  published  herein. 


Panel  I 


"Special  Problems  of  Operations  Research  in  Civil  Agencies 

of  Government" 


Chairman:  Mr.  Ezra  Glaser 

Health,  Education,  and  Welfare 

This  panel  was  to  review  communication 
problems  among  technical  operations  research- 
ers and  to  attempt  to  specify  those  factors 
which  limit  the  quality  and  extent  of  operations 
research  practice  in  the  civil  agencies. 

The  conference  proceeded  by  asking  those  in 
attendance  to  express  agreement  with  a  num- 
ber of  propositions,  each  of  which  postulated 
an  important  limiting  factor  for  the  quality  and 
extent  of  operations  research  practice.  Four 
classes  of  problems  and/or  limiting  factors 
were  identified  as  "severe,"  for  the  quality  or 
extent  of  operations  research  practice: 

1.  Difficulties  in  dealing  with  the  upper 
echelon  of  the  organization.  The  preva- 
lence of  preconceived  problems  and  precon- 
ceived answers  were  symptomatic  of  these 
difficulties.  Lack  of  understanding  by  the 
upper  echelon  of  what  operations  research 
is  and  how  it  might  usefully  be  employed 
in  solving  the  problems  of  the  organiza- 
tion. 

2.  The  training  and  quality  of  operations 
analysts. 

3.  The  difficulty  of  developing  measures  of 
effectiveness  in  civilian  agencies.  While 
there  has  been  some  success  in  construct- 
ing measures  of  effectiveness  in  business 
organizations  and  military  agencies,  there 
is  very  often  great  difficulty  in  finding 
means  of  measuring  the  effectiveness  of 
civilian  programs.  The  problem  is  how  to 
be  quantitative  without  by-passing  the 
principal  reasons  for  the  existence  of  the 
program. 

4.  Communications  among  operations  re- 
searchers. This  applies  both  to  the  tech- 
niques employed  and  to  results  obtained. 

There  was  little  support  for  the  proposition 
that  inadequacy  of  financial  resources  was  a 
serious  limiting  condition.  The  group  indicated 
a  belief  that  sufficient  funds  were  generally 
forthcoming  in  situations  where  there  was 
some  satisfactory  way  of  handling  the  problems 
noted  above. 


Relatively  little  weight  was  given  to  the  in- 
adequacies in  the  technical  literature,  which 
might  impede  teaching  or  the  upgrading  of 
quality  of  operations  researchers. 

Each  of  the  four  principal  obstacles  will  be 
discussed  in  turn. 

1.  Relation  to  Upper  Echelon  of  the  Orga- 
nization 

One  principal  and  frequent  problem  is  the 
indirectness  of  the  O.R.  staff's  organizational 
relationship  to  the  head  of  the  agency.  This 
means  that  the  staff  is  often  not  in  a  position  to 
watch  the  development  and  identification  of 
major  problems  from  the  viewpoint  of  the  agen- 
cy head,  and  therefore  not  in  a  position  to 
prepare  in  advance  for  studies  aimed  at  assist- 
ing in  reaching  important  decisions.  Experi- 
ence shows  that  such  operations  research  stud- 
ies require  substantial  lead  time  for  high- 
quality  results,  but  in  practice  this  time  is 
generally  unavailable.  The  likely  result  is  a 
hasty  and  ill-considered  effort  which  can  take 
but  little  advantage  of  the  potentialities  and 
resources  of  O.R.  techniques. 

2.  The    Training    and    Quality    of  O.R. 
Personnel 

It  was  pointed  out  that  technical  courses  in 
O.R.  that  might  prepare  analysts  for  govern- 
ment applications  leave  much  to  be  desired. 

It  was  suggested  that  an  interdepartmental 
and  consortium  approach  be  utilized  to  offer 
enough  kinds  of  courses,  for  different  clienteles 
and  describing  different  techniques  and  focuss- 
ing on  different  problem  applications  to  fill  out 
the  field  in  some  systematic  manner.  While  no 
single  institution  could  collect  enough  students 
or  assign  enough  specialized  faculty  for  such  a 
program,  it  seems  likely  that  a  group  of  educa- 
tional institutions  could  arrange  just  such  a 
spectrum  of  courses. 

Another  problem  area  is  the  recruitment  of 
college  personnel  for  operations  research  work. 
Generally  the  Government  is  not  competitive 
with  private  enterprise.  Competitiveness  re- 
quires sending  out  technical  recruiters  who 
are  able  to  discuss  the  technical  opportunities 
in  language  meaningful  to  recent  graduates  of 
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technical  courses.  Moreover,  recruiters  for  pri- 
vate enterprise  can  make  a  commitment  on  the 
spot  whereas  the  Federal  recruiter  can  only- 
point  to  the  long  and  tortuous  path  of  examina- 
tions, ratings,  etc.,  which  might  eventually  lead 
to  some  type  of  promotion.  While  these  con- 
straints are  consistent  with  Civil  Service  phi- 
losophy and  practice,  they  should  be  recognized 
as  placing  the  Federal  establishment  at  a 
serious  disadvantage  in  competing  with  private 
organizations  for  superior  students. 

The  group  recommended  more  emphasis  on 
sending  rusty  students  back  to  school,  renewing 
their  technical  abilities  and  extending  these  into 
operations  research  studies.  There  is  also  need 
for  a  series  of  shorter  courses,  especially  those 
directed  at  technique  in  the  modelling  process, 
with  use  of  "live"  exhibits  of  real  problems 
used  to  the  maximum  extent  possible. 
3.  Measures  of  Effectiveness 
This  is  one  of  the  most  difficult  problems  in 
the  useful  application  of  operations  research  to 
the  programs  of  civilian  agencies.  One  cannot 
measure  the  benefits  from  various  program 
elements  or  administrative  alternatives  by  their 
effects  on  "profits."  Nor  are  there  well  defined 
military  type  objectives.  The  difficulty  is  serious 
enough  as  regards  provision  of  current  services 
(health,  occupational  retraining,  vocational  re- 
habilitation, etc.)  but  is  an  order  of  magnitude 
more  severe  in  dealing  with  those  programs 
whose  benefits  lie  in  the  improvement  of  future 
services  (improvements  in  elementary  school 
education,  biomedical  research) . 

There  is  almost  a  total  absence  of  training 
materials  suitable  for  most  of  the  civilian  agen- 
cies of  the  Government,  which  communicates 
the  few  hard-won  currently  known  approaches 
to  the  problems  of  constructing  measures  of 
effectiveness. 

In  practice  this  problem  tends  to  be  linked 
with  the  relation  of  the  operations  research 
group  to  the  top  man  in  the  organization — in 
the  absence  of  opportunity  for  this  group  to 
formulate  and  inject  any  more  objective  meas- 


ure of  effectiveness,  the  evaluation  necessarily 
turns  out  to  be  an  expression  of  his  personal 
preferences. 

4.  Communication  Among  Operations  Re- 
searchers 

The  key  notion  here  seems  to  be  the  existence 
of  "natural  aggregates"  in  Government  pro- 
grams. For  example,  the  interests  of  the  Fed-  ! 
eral  Government  in  transportation  are  scat- 
tered over  many  agencies,  many  programs,  and 
many  objectives.  It  would  seem  desirable  for 
operations  researchers  who  are  working  on  the 
many  aspects  of  transportation  program  to  be 
in  communication  with  others  working  within 
the  same  aggregate. 

A  yet  broader  combination  can  be  obtained 
by  grouping  the  interests  of  transportation  and 
communication,  since  these  two  are  inter- 
changeable or  substitutable  in  some  situations. 

The  Planning-Programming-Budgeting  Sys- 
tem will  probably  provide  an  organizational  en- 
try into  these  natural  groupings  by  the  require- 
ments of  PPB  that  common  objectives  be  iden- 
tified and  put  into  some  rational  programming 
framework.  „ 

For  operations  researchers  whose  activities 
fall  into  some  natural  aggregate  (not  quite  as 
strictly  interpreted  as  Derek  Price's  Invisible 
Colleges),  it  would  seem  desirable  to  program 
a  certain  number  of  technical  colloquia  and  I 
seminars.  These  are  not  envisaged  as  meetings  j 
to  compile  lists  of  proposed  projects  or  on- 
going  research,  but  rather  as  solid  discussions  j 
in  considerable  depth  of  technical  problems,  at- 
tempts  to  use  various  techniques,  obstacles  met,  I 
tentative  results  achieved,  and  so  forth. 

One  question  that  arises  in  the  reconstruction 
of  budget  decision  making  categories  is  the  way  j 
in  which  the  Bureau  of  the  Budget  itself  would  j 
adapt  to  such  natural  aggregates  as  are  not  i 
now  expressed  in  its  own  organization,  and 
what  part  they  might  play  in  any  colloquia 
which  are  organized  in  behalf  of  some  substan- 
tial common  interests  among  the  Federal  agen- 
cies and  programs. 


Panel  II 

The  Summary  of  Proceedings  for  Panel  II  is  not  available. 
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Panel  III 


Broadening  the  O.R.  Competence  of  Mid-Careerists 


Chairman:  Mr.  Joseph  Lowell,  Jr. 

Civil  Service  Commission 

Panel  III  was  attended  by  13  participants, 
almost  all  of  whom  are  actively  engaged  in 
Operations  Research. 

Perhaps  the  most  significant  result  of  Panel 
Ill's  discussion  was  the  refinement  of  the  panel 
topic.  Participants  agreed  that  the  topic  was  too 
broad,  including  at  least  two  sub-problems 
worthy  of  quite  separate  treatment. 

The  first  problem  involves  the  identification, 
recruiting  and  training  of  those  persons  pres- 
ently in  non-O.R.  positions  who  have  the  poten- 
tial to  be  made  over  into  O.R.  practitioners. 

The  second  problem  involves  the  assistance 
and/or  training  which  non-technical  people 


need  in  order  to  understand  enough  about  O.R. 
to  know  when  and  where  to  turn  to  O.R.  prac- 
titioners for  help. 

(A  third  problem,  raised  but  not  extensively 
discussed,  involved  the  education  of  O.R.  prac- 
titioners in  the  art  of  communicating  their 
profession  and  its  techniques  in  lay  terms.) 

It  was  generally  agreed  that  each  of  the 
above  mentioned  problems  merited  serious  and 
separate  discussion.  Recommendations  were 
made  that  they  be  treated  individually  at  any 
follow-up  sessions.  There  was  general  agree- 
ment that  participants  in  such  discussions 
should  be  pre-registered  and  pre-selected  to 
avoid  the  situation  of  one  group  discussing  the 
problems  of  another,  unrepresented,  group; 
e.g.,  O.R.  people  deciding  independently  what 
non-O.R.  people  need  to  know,  or  vice  versa. 
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